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Summary 


This Workshop on Survey Methods in Education Research tool consists of a facilitator 
guide and workshop handouts. The toolkit is intended for use by state or district education 
leaders and others who want to conduct training on developing and administering surveys. 
The facilitator guide provides materials related to various phases of the survey develop- 
ment process, including planning a survey, borrowing from existing surveys, writing survey 
items, pretesting surveys, sampling, survey administration, maximizing response rates, 
and measuring nonresponse bias. It also contains a section on focus groups (as part of the 
survey development process or as a supplementary or alternative data collection method). 
The materials include a sample workshop agenda, presentation handouts, activities, addi- 
tional resources, and suggestions for adapting these materials to different contexts. 

The guide and materials were created for workshops conducted by the Regional Education- 
al Laboratory (REL) Midwest. These workshops were developed in response to district and 
state education leaders in the REL Midwest Region who were interested in building agency 
capacity to design and conduct high-quality surveys. 
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Introduction to the Workshop on Survey 
Methods in Education Research 


The Workshop on Survey Methods in Education Research guide and its accompanying 
slide decks constitute a set of resources that can be used to learn more about survey design 
and administration. The guide is intended to help researchers in state and local education 
agencies organize and conduct a training session on surveys for their staff, but others who 
are developing surveys can use the materials to facilitate workshops, guide a survey project, 
or ensure that they are adhering to best practices in survey methodology. The workshop 
is structured in eight modules that describe the steps of survey design and administration 
from planning to data collection. 

Why are surveys important in education research? 


The need for accurate data to inform sound decisions — whether to determine district 
and school policies, practices, or programs; identify the professional development needs 
of teachers; or justify grant funding — often requires state and local education agencies to 
collect information through surveys. Surveys are important in education research because 
they can provide quantitative descriptions of the characteristics, behaviors, and attitudes 
of students, teachers, principals, parents, district leaders, and other specific populations. 
Accurate data from a well-designed survey can be instrumental in guiding effective policy 
and program decisions. For example, surveys in education research can be useful in: 

• Assessing the needs of the parents of students who have individualized education 
programs. 

• Understanding the experiences of teachers during and after the implementation of 
a professional development workshop. 

• Tracking students’ attitudes toward going to college and their use of counseling 
services across multiple years. 

• Comparing the attitudes and experiences of school administrators who have expe- 
rienced different districtwide education reforms. 

• Measuring teacher supply and demand and assessing areas of future need. 

• Understanding barriers to parent involvement from parents’ viewpoints. 

Although it is relatively easy to create and administer a survey, it is not easy to create and 
administer a high-quality survey that will provide accurate findings that can inform crucial 
decisions. For example, consider the following scenario: 

An administrator within the Department of College and Career Readiness in a 
large urban district wants to collect data about students’ college plans to inform 
her about upcoming needs for counseling and programming and to help with 
college planning. Off the top of her head, the administrator types a few questions 
and distributes paper copies of the survey to principals at a districtwide meeting, 
asking them to pass the surveys along to their students, collect completed surveys, 
and send the surveys back to the district office. The administrator then compiles 
the answers from the completed surveys and writes a report. 

What questions would you ask the district administrator about her survey design and 
administration? How might she have amended her process to conduct a survey that would 
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provide accurate and actionable information? Consider the following questions, organized 
by the topics covered in this workshop: 

• Planning ♦ What were the goals of the research project? Was a survey the best 
method for reaching these goals? What questions did the administrator hope to 
answer? What population did the administrator want to talk about in her research? 
How did the principals know whom to give the surveys to and when to return the 
completed surveys? 

• Existing survey items ♦ Did the administrator examine existing surveys on col- 
lege-going plans for pre-existing, tested items? Are the items that she developed 
written in such a way that they will elicit reliable and valid data? 

• Writing items ♦ Were the items on the survey clear? Did the response options 
make sense? Were the questions and response options presented according to best 
practices? 

• Pretesting items ♦ Were the items tested to determine whether the survey respon- 
dents would interpret them as intended? 

• Sampling ♦ How were students selected to take the survey? Did the administrator 
determine and convey which students should take the survey? 

• Survey administration ♦ Was distributing a paper survey at a district meeting the 
best way to administer the survey to high school students? Was online data collec- 
tion an option? 

• Response rates ♦ What percentage of the surveys were actually completed and 
mailed in? What implications does the response rate have for the final report? Are 
the results representative of the population of interest? 

These questions are important to both survey methodologists and practitioners because 
the answers dictate the confidence that stakeholders can have in the results. 

What is included in these workshop materials? 


This guide and accompanying slide decks can be used for group or individual study to 
learn the basic principles and best practices of survey methodology in education research. 
The objective of a workshop conducted with these resources is to increase participants’ 
knowledge of the basics of survey research so that they can design and administer effective 
surveys. 

The materials are organized into eight modules: 

• Planning for a survey ♦ Covers survey goals, research questions, definitions of 
terms and constructs, survey reliability and validity, and alternatives to surveys. 

• Exploring sources of existing survey items ♦ Covers federal and other education 
survey sources. 

• Writing survey items ♦ Covers writing of clear items, open-ended and closed-ended 
items, response options, rating scales, and formatting and layout. 

• Pretesting survey items ♦ Covers expert review, cognitive interviews, focus groups, 
and field testing. 

• Sampling ♦ Covers sampling frames, random sampling, stratification, and census 
and other considerations. 

• Administering the survey ♦ Covers personal interviews, online surveys, and mailed 
surveys. 
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• Calculating response rates ♦ Covers calculating response rates, maximizing 
response rates, and measuring nonresponse bias. 

• Setting up a focus groups (as an alternative to a survey or to complement a 
survey project ). Covers purposes of focus groups, preparing protocols, selecting 
focus group members, moderator strategies, collection and analysis of data. 

Each module includes a basic overview of the topic, the purpose and goals of the session, 
logistical considerations, an annotated agenda, and participant materials (a slide deck, 
handouts, and sometimes an activity). A sample agenda, sample communication with par- 
ticipants, a needs-assessment questionnaire, and a postworkshop evaluation are in appen- 
dix A, and a glossary of survey-related terms is in appendix B. Slide decks for each module 
are available at https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4544. 

What is the source of the information presented in this workshop? 


The workshop materials are based on scholarly texts and literature on survey method- 
ology and best practices recommended by the American Association of Public Opinion 
Research. The content has been reviewed by representatives of state and local agencies 
in the Regional Educational Laboratory (REL) Midwest Region to ensure that the meth- 
odological topics (for example, constructing sampling frames and practices for increasing 
response rates) address survey design needs in the contexts of their agencies. In addition, 
each module was prepared in consultation with experts at the Center for Survey Methods 
at American Institutes for Research. The authors have conducted workshops on general 
survey knowledge for state and local education agencies, and their workshop materials 
have been adapted for use with REL Midwest and REL Southwest research alliance proj- 
ects to support the development of surveys for research agendas. 1 This guide incorporates 
feedback from participants and other lessons from those workshops. 

Although the theory of survey research methodology forms the basis of the workshop 
materials, the content illustrates the practical needs of local and state education stake- 
holders. For example, discussions about different types of survey respondents refer to school 
district staff and administrators, principals, teachers, students, and parents. Furthermore, 
specific strategies to adapt the modules to local education contexts and issues are provided 
in each module. 

Who should facilitate a workshop on survey methods? 


Ideally, the facilitator of a survey methods workshop will have expertise with survey 
methodology in education research through training or practice. However, with advance 
preparation, leaders of education research studies (for example, leaders of research and 
evaluation departments in state education agencies, technical assistance consultants, and 
education research professors) can successfully facilitate this workshop. 

Who should attend a workshop on survey methods? 


Staff tasked with developing, evaluating, or administering surveys can benefit from this 
workshop. The workshop is also appropriate for staff members who oversee individuals 
(including contractors) who develop and administer surveys for their organizations. 
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How should these materials be used to 
facilitate a workshop on survey methods? 


The main content of each module is found in the commentary paired with the slide decks. 
In addition, each module includes an agenda, background, handouts, and sometimes an 
activity. 

The eight modules can be presented together as an overview of survey methods during 
a one-day workshop or two half-day workshops; alternatively, modules can be presented 
separately and expanded to support a working group’s effort to plan or design elements of 
a survey project. For example, a working group might run the session on planning for a 
survey (module 1) and then work through the steps outlined in the session for a few weeks 
to finalize its plans. The group can then reconvene for the next two sessions about explor- 
ing sources of existing survey items (module 2) and writing survey items (module 3) to 
support the next stage of survey development. The workshop sessions can be strategically 
timed to support the group’s learning needs as it develops a survey and outlines a plan for 
administering it. 

The materials are flexible. The examples in each module can be replaced with exam- 
ples related to an ongoing survey topic, and the activities can be adapted or expanded to 
meet specific learning and planning goals. The guide and accompanying materials were 
designed to support a workshop or working group effort, but the materials can also be used 
for individual study. 
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Getting started: Planning the workshop 


This section presents the steps necessary to successfully plan and execute a workshop. 

Workshop goals 


Think about your organization’s current needs for a survey methods workshop. Does it 
need a workshop for a specific project or one designed to introduce staff members to survey 
methods? What would you like participants to know and be able to do by the end of the 
workshop? How will you assess whether the workshop was successful? 

Format of the workshop 


As the facilitator, you can set up the workshop to meet precise goals. You can structure 
the workshop according to whether you want to increase the capacity of your staff to 
develop and administer surveys in general or guide and support a specific survey project. 
Each module can even be used as a standalone presentation to review a topic (for example, 
survey administration). Format options for the workshop include: 

• A daylong introductory seminar on survey methods ♦ This option enables you and 
your staff to learn how to design, develop, and administer surveys for a new project 
or an existing project that is being improved, 

• A series of sessions over several weeks or months ♦ With this format, you could 
tailor the activities provided in the modules to develop a survey specifically for 
your organization. You could also split the workshop into a section on survey 
design and development and a section on survey administration training: 

o Option 1: Survey design and development workshop ♦ If you are interested 
in improving existing items on a current survey or developing items for a new 
survey, consider using modules 1-4. 

o Option 2: Survey administration training ♦ If you are interested in improving 
response rates or sampling or want to think about the best method to admin- 
ister an existing survey (for example, web or paper), consider using modules 1, 
5, 6, and 7, 

• Individual study ♦ Ask individuals to peruse the participant materials of a particu- 
lar subset of modules pertinent to your work and to develop a series of sessions on 
incorporating best practices into your current work. 

Participant list 


After you have researched and identified your organization’s needs, think about the target 
audience for your workshop. Your target audience may include department staff, research- 
ers, practitioners, students, and external partners. The choice of participants will depend 
on why you are conducting the workshop. For example, if you want to build your organiza- 
tion’s capacity for survey research, you may want to invite a large number of people, includ- 
ing individuals in other departments. If you have a specific task in mind, you may want to 
limit participation to individuals working on that project — in which case, be sure to have 
all key staff attend. Other considerations include the size of the workshop space and the 
number of facilitators and support staff. The workshop should be broadly accessible but of a 
manageable size. After you have prepared a potential participant list, ensure that you have 
up-to-date contact information for each person. 
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Needs assessment 


Consider sending a short survey to potential attendees about their prior experiences with 
survey methods- (You can adapt the “Preworkshop needs assessment survey” provided in 
appendix A.) Use the information from this survey to tailor your workshop to your partic- 
ipants. For example, if the participants frequently develop and administer surveys, select 
modules that can help them work through the development of a particular survey or trou- 
bleshoot such issues as low response rates or unreliable survey items. Conversely, if most 
of the participants have little to no prior experience with survey methodology, present 
an overview of concepts, giving special attention to how each concept might be useful in 
their work. Finally, if the participants represent a mixed-abilities group, consider breaking 
them into small groups throughout the workshop so that they can learn from each other as 
they work on discrete tasks of differing skill levels. 

Some additional tips: 

• Allow enough time for participants to acknowledge receipt of the preworkshop 
needs assessment survey, complete it, and return it. Also, build in time to review 
their responses and then tailor the workshop accordingly. Allowing at least four 
weeks between sending the preworkshop survey and completing design of the 
workshop is suggested. 

• Increase buy-in for the workshop by asking members of your staff with experience 
in a particular module topic area to facilitate that portion of the workshop. 

• Consider developing a small graph (for example, a bar graph with the number of 
individuals from each department) to use in your introduction in order to give 
participants a sense of who is in the room. 

Logistics 


Important components of a successful workshop are planning and communication. Use 
the following checklist to ensure that you are addressing all logistical concerns: 

Invite participants 

• Send the initial invitation. 

• Incentivize participation, if appropriate. 

• Send out preworkshop needs assessment survey. 

• Tailor workshop to skill level of participants. 

• Send a reminder to the participants three days prior to the workshop. 

Reserve a room 

• Ensure that the workshop space is large enough. 

• Ensure access to a projector and a computer or laptop connection. 

• Test the technology prior to the workshop date. 

Gather materials 

• Handouts (agendas, slide notes, worksheets). 

• Facilitator guide. 

• Slide deck. 

• Session materials (markers, poster board, pens). 
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Other considerations 

• Food and drink. 

• Parking. 

• Accommodations for attendees with special needs. 

Agenda for session kickoff 


Regardless of which modules you use, address the following agenda items at the beginning 
of the workshop (allow 15 minutes): 

• Introduce yourself. 

• Answer any logistics questions (for example, schedule and location of restrooms). 

• Explain the goals of the workshop (for example, “Why have we convened this 
group?”). 

• Briefly review the agenda. (A sample one-day agenda is provided in table A1 in 
appendix A.) 

• Ask participants to introduce themselves and quickly describe their interests in 
the workshop. 


Postworkshop evaluation 


After the workshop administer a postworkshop evaluation survey to assess whether the 
participants found the workshop useful. You can use the “Postworkshop needs assessment 
survey” provided in appendix A or adapt it by including items more specific to your goals 
or local context. If possible, administer the survey in person at the end of the workshop. 
Synthesize the responses into a format that you can use to gain a better understanding of 
participants’ experiences. Tips for assessing responses are as follows: 

• Count how many respondents answered in each response category for each item. 

• Summarize the responses to open-ended items. 

• If you are planning to offer the workshop again, conduct a participant focus group 
to assess the strengths and weaknesses of the workshop. 
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Module 1: Planning for a survey 


Overview 


Duration: 45 minutes, including 15 minutes for activity Id 
Slides: Module 1 slide deck 
Handout: Handout Id: Planning for a survey 
Materials needed: 

• One copy of slides and handout Id per participant- 

• Writing tools. 

• Projector and computer. 

Agenda 


• Is a survey the right approach? 

o Deciding to conduct a survey, 
o Considering other approaches. 

• Planning for a survey. 

o Setting research goals and research questions, 
o Defining constructs, 
o Identifying subgroups of interest, 
o Determining the unit of analysis. 

o Identifying types of survey respondents best able to provide accurate data. 

• Considerations. 

o Ensuring reliability, 
o Ensuring validity. 

• Activity 1.1. 


Background 


The first module walks participants through the process of planning a survey: setting 
research goals, composing research questions, defining the important topics and constructs 
to be measured, and thinking about the population that will be surveyed. Although it is 
tempting to skip ahead to writing survey items, having a clear plan that is agreed on by all 
stakeholders is a critical first step. The goal of this module is for participants to understand 
the importance of this phase of survey development. 

Strategies for adapting module 1 to your context 


The planning module can be broken down into multiple sessions to help stakeholders con- 
sider different aspects of a survey that is in the earliest stage of development. For example, 
one session might focus on developing goals and research questions, and the next might 
focus on identifying the subgroups of interest and the survey respondents. Consider whether 
you want participants to respond to questions about a hypothetical survey scenario, a sce- 
nario provided by you that is relevant to your local context, or an actual survey they are 
working on. If the participants are collaborating on an actual survey, use the activity to 
define research questions and other details for that survey. Have the participants complete 
the activity in teams or as a whole to produce something that is relevant to their work. 
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Slides and notes for module 1 


Slide 1 


Slide 2 



Module 1 
presentation 

Planning 
for a 

SURVEY 


Outline 


• Is a survey the right approach? 

• Planning for a survey. 

• Considerations. 

• Activity. 


Slide 3 


Outline 


• Is a survey the right approach? 

• Planning for a survey. 

• Considerations. 

• Activity. 


Slide 4 



A survey can produce quantitative descriptions of the 
characteristics and attributes of a population. 

Surveys are appropriate data collection tools for many purposes 
but are not the best method for every research project. 

Other methods can be considered before deciding to create a 
survey. 
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Slide 5 


Slide 6 


Slide 7 


Slide 8 



The four primary options for data collection, other than surveys, 
are semi-structured interviews, focus groups, observations, and 
analyzing existing data. 


Interviews 


Interviews are suitable for investigating complex topics without 
well-defined research questions. 



Focus groups 




When to use focus groups 
instead of a survey 


Focus Groups Surveys 


Focus groups may reveal interesting perspectives about a topic 
through group interaction. The discussion among participants 
about a topic can provide a more rich description of their views 
and experiences than a set of individual interviews or surveys with 
the same people. 

Focus group methodology is described in module 8. 


In some cases, a focus group study may be more appropriate than 

a survey study. A focus group is preferred if: 

• The important details and questions surrounding the topic are 
not well known or well defined. 

• The topic is complex, and you want to understand people’s 
views and positions. 

• You do not need a representative sample of a population to 
meet the goals of the project. 

• You think group interaction may reveal interesting perspectives 
about the topic. 

• The issues or response categories are not clear. 


• It is important to have individuals with a diverse range of 
opinions interact with each other. 


• It is important to hear from and be responsive to multiple 
stakeholder groups. 

• You are interested primarily in individuals’ feelings and 
perceptions. 


• You are trying to understand individuals’ or organizations’ 
processes. 
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Observations 


Slide 9 


Slide 10 


Slide 11 




• Is a survey the right approach? 

• Planning for a survey. 

• Considerations. 

• Activity. 


Observations can be used to collect data about behaviors or 
practices (for example, teacher-child interactions) that might not 
be reported accurately in a survey. 


Analyzing existing data can be a cost-effective way to explore 
relationships between variables in administrative data files. 


Slide 12 


Research goals 


Think about 
how the 
results will 
be used. 



What are your goals? What do you want to learn from the survey 
results? 

• Describe current practices or behaviors. 

• Understand attitudes. 

• Evaluate the outcomes of a program or an initiative. 

Consider the following: 

• Available funds. 

• Timeframe. 

• With whom will the findings be shared? 

• What actions or decisions will the survey results inform? 
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Slide 13 


Slide 14 


Slide 15 


Slide 16 



Clarify your 
research goals. 



Align research questions 
to your research goals. 




Define each construct in 
your research questions. 



When specifying the goals of the survey, provide a clear 
explanation of why the survey is being created, whom the results 
will be shared with, and how the results will be used. 

Clarify the actions or decisions you want the results of the survey 
to inform. Why do you want this information? For example: 

• Should we spend more resources to encourage non-English- 
speaking parents to participate in kindergarten orientation 
activities? 

• What is the best way to allocate resources for increasing school 
safety at our high schools? 

Consider all stakeholders when setting the goals for a survey. 
Write research questions, describing what you want answered by 
your survey data. Research questions should describe what you 
want to know and about whom. Consider the following examples: 

• What barriers to school involvement are non-English-speaking 
parents experiencing? 

• How physically safe do high school students feel at their 
school, and how does this relate to school safety measures? 


Evaluate each research question against the survey’s research 
goals. Resist the temptation to add topics and items that do not 
directly serve the stated purpose. 

One strategy to help clarify the plans for the survey is to envision 
writing the survey findings. What do you want to say in your 
report? For example: 

• “ percent of parents of English learner kindergartners and 

percent of parents of non-English learner kindergartners 

attended kindergarten orientation activities.” 

• “ percent of high school students reported witnessing 

violent crime in the school building or on school grounds in the 
past year.” 

Be mindful of time and budget when establishing what the survey 
will accomplish. Spend time preparing and gaining consensus on 
the goals and research questions. These important documents 
will guide every decision along the way. 

A construct is an idea or a characteristic that may be thought of 
in different ways (for example, family engagement, leadership, or 
highly qualified). Explicitly defining the constructs for a survey is 
important at the planning stage. To ensure consideration of all 
important elements of a survey, survey developers should look to 
the literature, existing surveys, experts, stakeholders, and their 
own prior knowledge and experiences. 

Be specific when defining constructs. For example, if district 
administrators are interested in teacher perceptions of family 
engagement, the survey might cover topics such as attendance 
at school events, frequency of communication, and shared 
decisionmaking. It will be important to reach a consensus on what 
each construct means and which aspects are important to include 
on a survey. 
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Slide 17 


Slide 18 



Example research 
question: 


How is participation in 
extracurricular activities 
related to our high school 
students' academic 
engagement? 


Example research question: How is participation in extracurricular 
activities related to our high school students’ academic 
engagement? 

• What do you want to know about participation in extracurricular 
activities? 

• What is the type of activity (for example, social, academic, 
sports, or club)? 

• What is the type of participation (for example, leadership role)? 

• How many different activities do students participate in? 


• How much time per week do they spend in activities? 


• How many school years have students been involved in the 
activity? 


There are multiple ways to define and measure the construct 
academic engagement. 

• Review academic literature. 


• Convene a panel of experts in the field. 

• Consult existing surveys that measure the construct. 

Constructs such as this are typically measured with a set of 
items. 



Identify subgroups of 
interest. 



Within the population you want to describe (for example, students, 
teachers, or schools), across what characteristics do you want to 
compare the survey findings? Consider the following: 

• Student: gender, race/ethnicity, grade level, English learner 
status, special education status, and eligibility for the school 
lunch program. 

• Teacher: gender, race/ethnicity, grade level, license area, and 
years of experience. 

• School: grade levels, enrollment, percentage of students from 
minority backgrounds, and school type. 


Determine what subgroup comparisons would be of interest. 
These subgroups should be chosen based on a perceived 
relationship with the survey’s topic of interest. For example, 
if a survey is about schools’ implementation of response to 
intervention statewide, it should include items that relate to 
school characteristics that might affect implementation (for 
example, school size, school type, grade levels, and the number 
of qualified personnel). 


Determining the appropriate subgroups is important so that the 
findings can be presented in light of these different subgroups. 

In some cases, the subgroup characteristics can be linked 
to the survey responses by linking identification numbers to 
administrative files or other databases and will not need to be 
included as additional items on the survey. For example, if school 
identification numbers are linked to school surveys, information 
about the school that can be obtained through records, such as 
school size, would not need to be included as survey items. 
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Slide 19 


Slide 20 




Identify who 
will provide 
the most 
accurate data 
for each 
construct. 



The unit of analysis is the set of people, programs, or institutions 
the researcher wants to make inferences about. For example, you 
might want to make statements about the following groups: 

• Students (for example, 30 percent of the students walk to 
school). 

• Families (for example, 30 percent of the families have a child 
who walks to school). 

• Schools (for example, 30 percent of the schools have a majority 
of students who walk to school). 

• Districts (for example, 30 percent of the districts have a 
majority of students who walk to school). 

As another example, if the goal of the survey is to describe the 
characteristics of afterschool academic programs, the unit of 
analysis is the program. The program director, teacher, and 
student data will be structured so that the findings are about 
the program (for example, the percentage of programs that 
provide transportation, the percentage of programs with at least 
one highly qualified teacher, or the average number of students 
enrolled in a program). 

It may be useful to consider multiple units of analysis for a survey 
about afterschool programs (for example, the percentage of 
schools with a program or the percentage of parents in schools 
with a program who are aware of the program). 

Ensure that whom the survey is administered to and the unit 
of analysis align with the data structure needed to answer the 
research questions. 

Carefully identifying who can best provide the information 
needed will help determine the design of the study. For 
instance, if workshop participants are interested in measuring 
kindergarteners’ adjustment to school, consider whether 
kindergartners themselves, teachers, or parents will be best 
suited to provide that information. This decision has implications 
for how the survey will be developed and administered. 

Example research question: Flow physically safe do high school 
students feel at their school, and how does this relate to schools’ 
safety measures? 

• High school students can report how safe they feel. 

• Who is best able to report on school safety measures (the 
principal, teachers, a trained observer, or security staff)? Are 
multiple respondents needed to get a complete picture? 

Example research question: What professional supports are 
available and used by elementary school math teachers? 

• Teachers can report what supports they know about and use. 

• Who will give the most accurate data about what supports are 
available? 

The unit of analysis and the data source are not always the same. 
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Slide 21 



Careful 
planning 
is essential 
for 

developing 
a survey. 



In summary: 

• Clarify the purpose of the survey. 

• Write research questions. 

• Carefully define important constructs. 

• For drawing comparisons, determine the important subgroups 
of interest. 

• Determine the most knowledgeable respondent or respondents 
for each construct. Decide if multiple respondents are needed 
to meet the survey goals. 


Slide 22 


Outline 


• Is a survey the right approach? 

• Planning for a survey. 

• Considerations. 

• Activity. 
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Reliability is the 
consistency with which 
a set of items measures 
an attribute. 


(Pastor, 2013) 



Validity is the extent to 
which the inferences 
made based on survey 
responses are accurate 
and meaningful. 

(Pastor, 2013) 


Reliability and validity are important considerations at multiple 
stages of survey development. 

Reliability is the consistency with which a set of items measures 
an attribute. A reliable item is interpreted the same way by 
all respondents and adequately covers the elements of each 
construct, thereby producing stable results about those 
constructs. For a single item, reliability can relate to how item 
wording ensures that a respondent will understand the item the 
same way at different points in time. It also can relate to the 
extent to which different respondents understand an item to have 
the same meaning. With a group of items measuring the same 
construct, reliability can relate to the extent to which the items 
are answered consistently (for example, if item 1 is answered yes, 
item 2 also should be answered yes). 

Survey items that are not reliable do not produce consistent 
responses at different points in time and across respondents. 
This inconsistency introduces measurement error (Pastor, 2013). 
Validity is the extent to which the inferences made based on 
survey responses are accurate and meaningful. A valid survey 
instrument provides results that accurately reflect the measured 
constructs and the target population as intended by the 
individuals who designed the survey and who will interpret the 
results. 

A valid item measures what the researchers intend it to measure. 
For example, a question on student perceptions of school safety 
must accurately measure student perceptions of safety in the 
school and not safety elsewhere. 

Survey items that are not valid do not capture the true meaning of 
the construct and therefore do not measure what the researcher 
intended (Pastor, 2013). 
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Slide 25 


Outline 


• Is a survey the right approach? 

• Planning for a survey. 

• Considerations. 

• Activity. 
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Activity 1: Planning for a 
survey 



Slide 27 


Additional resources 


Kendziora, K., & Boccanfuso, C. (2011). School climate survey series: 

Survey management [Survey webinar series for the National Center on 
Safe Supportive Learning Environments]. Retrieved April 1, 2016, from 
http://safesupportiveschools.ed.aov/index.php?id=9&eid=6 . 

Shindler, J. (2011). School climate survey series: Survey 

development [Survey webinar series for the National Center on Safe 
Supportive Learning Environments]. Retrieved April 1, 2016, from 
http://safesupportiveschools.ed.aov/index.php?id=9&eid=8 . 

For more information, please visit the following websites: 

• The American Association for Public Opinion Research: 
http://www.aapor.org/ . 

• Public Opinion Quarterly: https://pog.oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.org/content/current . 


Slide 28 


Reference 


Pastor, D. A. (2013, April). Considerations in evaluating, selecting and 
developing data sources and measures. IES webinar for Regional 
Educational Laboratories. 
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Activity 1.1: Planning for a survey 


This activity consists of addressing the questions in handout LL Choose one of the three 
options described here: 

During the presentation ♦ Having participants complete the activity during the slide pre- 
sentation makes for a more interactive session. You could insert the questions in the fol- 
lowing order: 

• After slide 15: Questions 1 and 2 (purpose and research questions) 

• After slide 18: Question 3 (constructs) 

• After slide 19: Questions 4 and 5 (unit of analysis and subgroups with similar 
characteristics) 

• After slide 20: Question 6 (respondents) 

After the presentation ♦ Participants can work individually or in groups to answer ques- 
tions about a hypothetical survey scenario or about an actual survey that their team is 
developing. For a team developing a survey, the activity can guide important decisions 
about the survey’s purpose and scope, which will require substantive discussion. 

As a take-home activity ♦ If the participants will complete the activity after the session, 
provide clear expectations for when and how they should deliver it, which resources they 
can use if they have questions, and what form of follow-up will take place (for example, 
another session to review answers). This option will work best if you are presenting the 
modules individually and if participants are all collaborating on a project that is directly 
related to their work. 
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Handout 1.1: Planning for a survey 


Think about a survey project you will design or update in the near future- 


L What are the goals for the survey? 


2- What are your research questions? 


3- What are the constructs of interest? How will you define important constructs? 


4- What is the unit of analysis? 


5- What are the subgroups of interest? 


6- Which types of respondent will provide the most accurate data for your survey? 
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Module 2: Exploring sources of existing survey items 


Overview 


Duration: 30-45 minutes 
Slides: Module 2 slide deck 

Handout: Handout 2d: National Center for Education Statistics surveys. 

This handout provides a table of 18 National Center for Education Statistics surveys that 
are sources for existing survey items, including each survey s name, topics, purpose, respon- 
dents, years administered, and a link to the survey. 

Materials needed: 

• One copy of slides and handout 2.1 per participant. 

• Writing tools. 

• Projector and computer. 

• Participant computers (optional). 

Agenda 


• Why use existing surveys? 

• About existing sources. 

• Sources. 

o National Center for Education Statistics. 

o National Center for Education Evaluation and Regional Assistance, 
o Other resources. 


Background 


High-quality survey items can be costly and time intensive to develop, test, and revise so 
that they are both reliable and valid (see module 1 for information about these concepts). 
Using existing survey items or sets of items is not only more efficient, but it has the addi- 
tional benefit of producing results that can be compared with results from other studies 
that use the same items. Several federal centers house survey items developed for education 
research. Two centers used in this module are the National Center for Education Statis- 
tics and the National Center for Education Evaluation and Regional Assistance, but other 
federal centers, such as the National Center for Health Statistics, post surveys online for 
survey developers. 

The goal of this module is to introduce sources of items that are included in large-scale 
surveys and encourage participants to look for existing validated items before writing their 
own items. 


Strategies for adapting module 2 to your context 


• If your department or the participants in your workshop are interested in develop- 
ing a specific survey, we highly encourage adapting this module to include examples 
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of that topic in lieu of the examples provided- Search tools on the websites of 
the National Center for Education Statistics (http://nces-ed.gov) and National 
Center for Education Evaluation and Regional Assistance (http://ies.ed.gov/ncee/) 
can help you locate appropriate surveys and survey items for your group. In addi- 
tion, handout 2.1, which includes information about the topics covered by many 
National Center for Education Statistics surveys, may be a helpful reference for 
identifying relevant surveys. 

• Allow participants a few extra minutes to review handout 2.1 and highlight the 
sources they are interested in learning more about. If participants have access to 
Internet-connected devices, encourage them to review the surveys for items that 
are pertinent to their work and have them discuss what they find. 

• Consider downloading the entire survey, providing participants with copies, and 
asking them to circle or highlight items that would be useful for a survey currently 
under development. 
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Slides and notes for module 2 


Slide 1 


Module 2 
presentation 

Exploring 
existing 
item sources 


Slide 2 


Outline 


• Why use existing survey items? 

• About existing sources. 

• Examples of sources. 



Slide 3 


Outline 


• Why use existing survey items? 

• About existing sources. 

• Examples of sources. 
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Why use existing survey 
items? 


When developing a survey, a good strategy is to look at other 
tested surveys on a similar topic and consider borrowing or 
adapting existing items. 


Why? 
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Why use existing survey 
items? 


Slide 5 


Time. 



Slide 6 


Why use existing survey 
items? 


And money (if hiring someone to develop survey items). 




Slide 7 


Slide 8 


Why use existing survey 
items? 



Outline 


• Why use existing survey items? 

• About existing sources. 

• Examples of sources. 


Using existing survey items not only saves time and money but 
also allows you to measure constructs in the same way that 
others in the field have measured them. 

Although writing new survey items may seem easy, if done 
correctly, it is actually a difficult and time-consuming process. 

New survey items need to be tested to ensure that they are 
capturing the information accurately and often require several 
revisions. 

Not all existing survey items have been tested for reliability, 
validity, and other important psychometric properties. Whenever 
possible, use trusted sources that test survey items for reliability 
and validity. Some of these sources are cited here and in handout 
2.1. When in doubt, check the documentation of existing survey 
items for details on their reliability and validity. 

Ensure that you have permission to use an item before including it 
on your survey. 
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Existing surveys 


Slide 9 


"NCES 

National Center for 
Education Statistics 



This discussion highlights sources from which you can find 
existing items and answer scales. The sources are from two 
centers within the Institute of Education Sciences: 

• National Center for Education Statistics (NCES). 

• National Center for Education Evaluation and Regional 
Assistance (NCEE). 


Slide 10 


Slide 11 


Considerations: 
Borrowing items 


How was the item 
asked originally? 


Who answered the 
question originally? 



Before borrowing an existing item, consider the following: 

• What was the mode of the survey (for example, phone interview, 
computer, or paper-and-pencil survey)? The mode can affect 
how an item is asked. For example, a survey administered on 
the phone by an interviewer allows the survey respondent to 
clarify any confusing aspects of an item. If you borrow an item 
designed to be asked over the phone to use in a paper-and- 
pencil survey that respondents must answer on their own, 
ensure that important terms and constructs are defined clearly. 

• Who answered the original survey (for example, students, 
parents, or teachers)? Different groups may think and talk 
about the same topic differently, so carefully consider how 
an item may need to be adapted for the type of respondent 
answering the survey. 


• One further consideration is whether the borrowed item is 
publicly available or if permission is needed. Always provide 
citations for borrowed items. 


NCES surveys 



You can search NCES surveys online (http://nces.ed.gov) to 
identify items or scales for your survey. Survey experts have 
evaluated items and scales on these surveys to determine that 
they are capable of yielding valid and reliable data for a given 
population. 

Check survey-specific documentation, such as data file user 
manuals, methodology reports, and psychometric reports, for 
detailed information on reliability and validity. 


Slide 12 


NCEE reports 




You can also search NCEE reports that summarize and detail 
survey item sets related to specific constructs, such as school 
readiness or student engagement (http://ies.ed.gov/ncee/; click 
on “Publications & Products”). Many of the reports detail survey 
items and sets of survey items used in evaluation studies. 
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Slide 13 


Topics 



NCES and NCEE administer surveys on a variety of topics, such 
as: 

• Parent involvement. 

• School climate. 

• School readiness. 


Slide 14 


Slide 15 



Outline 


• Why use existing survey items? 

• About existing sources. 

• Examples of sources. 


NCES and NCEE also administer surveys to a variety of 
respondents, such as: 

• Teachers. 

• Parents. 

• School administrators. 

• Students. 


Slide 16 


NCES schools and 
staffing survey 

33. In your FIRST year of teaching, how well pr 

* If you are in your first year of leaching, plea 

5 

THIS school yt 

m 


=r o,< “ 

- D 

2 □ 

’ D 

- D 

b - “ e,vo,l " ,,mc,ional « 

i □ 

2 □ 

2 □ 

4 □ 

c. Teach your subject matter? ... 

, □ 

> □ 

, □ 

4 □ 

d - “" Cl_ - 

. □ 

2 □ 

. □ 


.. Assess students? 

i □ 

2 □ 

3 □ 

4 □ 

, — .. 

, □ 

2 □ 

3 □ 

4 □ 

•=“= - 

i □ 

i □ 

2 □ 

2 □ 

3 □ 

3 □ 

4 □ 

4 □ 


The NCES Schools and Staffing Survey includes items about 
teacher perceptions and practices (http://nces.ed.gov/surveys/ 
sass/). 


Slide 17 


Parent and family 
involvement survey 


Factors Affecting Parent and Family Participation in School and Parent Support for the School. This 
section is asked of children in grades K-12 (PFI only). 


PLINTRO. Now let’s talk about things that may affect your family's involvement with (CHILD)'s school. 

PL1. How often does the school hold meetings during times that fit your work and family schedules? 

Would you say... 


The NCES National Household Education Surveys include survey 
items for parents about many education topics (https://nces. 
ed.gov/pubsearch/pubsinfo. asp?pubid=2013028rev). See also 
http://nces.ed.gov/nhes/. 


FPMTGWRK 


Sometimes, or 2 
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Slide 18 


Education longitudinal 
study of 2002 



The NCES Education Longitudinal Studies include surveys of 
secondary teachers and students. An example of a set of items 
for a student survey about school climate can be found at http:// 
nces.ed.gov/surveys/els2002/. 


Slide 19 


NCEE reports: Student, teacher, 
and classroom measures 


Compendium of Student, 
Teacher, and Classroom 
Measures Used in NCEE 
Evaluations of Educational 
Interventions 


The NCEE report Compendium of student, teacher, and classroom 
measures used in NCEE evaluations of educational interventions 
is a two-volume compilation and review of 94 student, teacher, 
and classroom outcome measures used in lES-funded 
evaluations between 2005 and 2008 (http://ies.ed.gov/ncee/ 
pubs/20104012). 


Volume I: Measures Selection 
Approaches and Compendium 
Development Methods 
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NCEE reports: 
Student engagement 



Measuring 
student 
engagement in 
upper elementary 
through high 
school: a 
description of 
21 instruments 



The NCEE report Measuring student engagement in upper 
elementary through high school: A description of 21 instruments 
reviews 21 instruments that measure student engagement 
through the later years of elementary school through high school. 
Scales (that is, a set of survey items, typically three or more, 
that measure a construct) include student reports, teacher 
reports, and observation (http://ies.ed.gov/ncee/edlabs/regions/ 
southeast/pdf/REL_2011098.pdf). 
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NCEE reports: 

Character education 

NCEE2009-006 U. S. DEPARTMENT OF EDUCATION 


Survey of Outcomes 
Measurement in Research on 
Character Education Programs 


ies 


NATIONAL CENTER fob 

EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 


I n it i I u t • el Education Sci.nc. 


The NCEE report Survey of outcomes measurement in research on 
character education programs reviews the constructs, frameworks, 
and measures of character education programs. Examples include 
student outcomes, such as critical thinking and decisionmaking, 
attitudes, and various behaviors (http://ies.ed.gov/ncee/ 
pdf/2009006.pdf). 
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NCEE reports: 
School readiness 



The NCEE report A review of methods and instruments used in state 
and local school readiness evaluations reviews the methods and 
instrumentation used in school-readiness analyses and program 
evaluation (http://files.eric.ed.gov/fulltext/ED497789.pdf). 
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Slide 23 


NCEE reports: Resiliency and 
youth development 



Measuring 
resilience 
and youth 
development: 
the psychometric 
properties of 
the Healthy 
Kids Survey 


The NCEE report Measuring resilience and youth development: 
the psychometric properties of the Healthy Kids Survey describes 
the psychometric properties of resilience measures in the 
resilience and youth development component of the Healthy 
Kids Survey (http://ies.ed.gov/ncee/edlabs/projects/project. 
asp?ProjectlD=84). 
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Other resources: 
Social-emotional learning 


Compendium of Preschool Through Elementary School 
Social-Emotional Learning 
and Associated Assessment Measures 


The NCEE report Compendium of preschool through elementary 
school social-emotional learning and associated assessment 
measures focuses on 5- to 10-year-olds and outlines a variety of 
scales that measure social-emotional learning (http://www.isbe. 
state, il.us/learningsupports/pdfs/sel-compendium. pdf). 


Susanne A. Denham 
George Mason University 

Peter Ji 

University of Illinois at Chicago 

Bridget Hamre 
University of Virginia 
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Other resources: 
School climate 



The National Center on Safe Supportive Learning Environments 
website (http://safesupportiveschools.ed. gov/index. php?id=133) 
has links to approximately 50 school climate surveys or surveys 
that include school climate items. Some of these surveys are 
copyright protected, so check with the publisher before using any 
items. 


Slide 26 



Q-Bank is a resource that provides information about survey items 
included on many federal surveys (http://wwwn.cdc.gov/qbank/ 
Home.aspx). It offers access to thousands of items that appear 
on federal surveys. Users can search items by keywords, topic, 
agencies that tested an item, and specific surveys where the item 
appears. Searches return links to reports that describe findings 
related to the items tested (for example, cognitive interview 
results). Typically, searches will link to reports related to items 
tested for a particular survey. Q-Bank is not a comprehensive 
system of every item on all federal surveys. 


Slide 27 



The Common Education Data Standards standardizes the way 
that agencies collect basic information so that comparisons can 
be made across local, state, and federal agencies. The website 
has standardized definitions of common variables (https://ceds. 
ed.gov/). 
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Many existing item sources are available. In addition to the 
sources cited earlier, there are other federal surveys that include 
survey items, about disabilities and student risk-taking behavior. 
Examples include the American Community Survey administered 
by the U.S. Census Bureau and the Youth Behavior Risk Survey 
administered by the Centers for Disease Control and Prevention. 

Finally, ensure that you have permission before using existing 
items. Check technical manuals to ensure that the item measures 
what you want to measure. 
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Additional resources 


Shindler, J. V. (2011). Survey development [Survey webinar series for 
the National Center on Safe Supportive Learning Environments]. 
Retrieved April 1, 2016, from 

http://safesupDortiveschools.ed.gov/index. php?id=9&eid=8 . 

For more information, please visit the following sites: 

• The American Association for Public Opinion Research: 
http://www.aapor.ora/ . 

• Public Opinion Quarterly: https://pog.oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.org/content/current . 
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Reference 


Pastor, D. A. (2013, April). Considerations in evaluating, selecting, and 
developing data sources and measures. IES webinar for Regional 
Educational Laboratories. 
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Handout 2.1: National Center for Education Statistics surveys 


1 Study 

Topics 

Purpose 

Respondents 

Years 

Survey link | 

National 

Adult education, parent 

• Provides reliable estimates of the U.S. 

Parents or 

1991, 1993, 

http://nces. 

Household 

Education 

Surveys Program 

and family involvement 
in education, early 
childhood program 
participation, school 
readiness, after school 
programs and activities, 
civic involvement, house- 
hold and library use, 
and school safety and 
discipline 

population regarding specific educa- 
tion topics and conducts repeated 

measurements of the same education 
phenomena at different points in time. 

guardians 

1995, 1996, 
1999, 2001, 
2003, 2005, 
2007, 2012 

ed.gov/nhes/ 

Early Childhood 

Schooling and perfor- 

• Focuses on children’s early school 

Teacher, school 

1998-99, 

http://nces. 

Longitudinal 

mance in the elementary 

experiences beginning with kindergar- 

administrator, 

1999-2000, 

ed.gov/ecls/ 

Study — 
Kindergarten 
Cohort of 1998 

and middle grades; 
status and transition; 
and interaction of school, 
family, and community 

ten and following them through middle 

school. 

• Provides descriptive information on 
children’s status at entry to school, 
their transition into school, and their 
progression through grade 8. 

• Enables researchers to study how fam- 
ily, school, community, and individual 

factors are associated with school 
performance. 

parent, child 

2002, 2004, 

2007 


Early Childhood 

Children’s growth and 

• Provides national data on children’s 

Parent, 

2001-02, 

http://nces. 

Longitudinal 

development, children’s 

status at birth and various points 

father-specific; 

2003-04, 

ed.gov/ecls/ 

Study— Birth 

transitions, children’s 

thereafter; children’s transitions to 

teacher, early 

2005-06, 


Cohort 

health status, father 
involvement, and school 

readiness 

nonparent care, early education pro- 
grams, and school; and their experienc- 
es and growth through grade 5. 

• Enables researchers to test hypothe- 
ses about the effects of family, school, 
community, and individual variables on 
children’s development, early learning, 
and early performance in school. 

education care 
provider 

2007 


Early Childhood 

Child growth and devel- 

• Provides comprehensive and reliable 

Parent or 

2010-11, 

http://nces. 

Longitudinal 

opment, school prog- 

data that can be used to describe and 

guardian, 

2011-12, 

ed.gov/ecls/ 

Study — 

ress, school transition, 

better understand children’s develop- 

child, school 

2013, 2014, 


Kindergarten 

Class of 

2010-11 

socioemotional develop- 
ment, home education, 
curriculum, teacher 
qualifications, and before 

and after school care 

ment and experiences in the elemen- 
tary grades plus how early experiences 
relate to children’s later development, 
learning, and experiences in school. 

• Enables researchers, policymakers, 
and educators to study how student, 
home, classroom, school, and com- 
munity factors at various points in a 
child’s life relate to cognitive, social, 
and emotional development. 

administrator, 

teacher 

2015, 2016 
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1 Study 

Topics 

Purpose 

Respondents 

Years 

Survey link 1 

National 

Assessment: math, 

• Continuously monitors the knowledge, 

Students, 

1969-1981, 

http://nces. 

Assessment 

reading, writing, arts, sci- 

skills, and performance of the nation’s 

teachers, 

1984, 1986, 

ed.gov/ 

of Educational 

ence, civics, geography, 

children and youth. 

and school 

1988, 1990, 

nations 

Progress 

technology and engineer- 
ing literacy, U.S. history, 
economics, document 
literacy, computer compe- 
tence, citizenship/social 
studies; and the Index of 

Basic Skills 

• Provides objective data on student 
performance at the national, regional, 
and state levels. 

principals 

1992, 1994, 

1996-2017 

reportcard/ 

National 

Assessment: prose liter- 

• Evaluates the English language literacy 

Adults age 16 

1992, 2003 

http://nces. 

Assessment of 

Adult Literacy 

acy, document literacy, 
quantitative literacy 

skills of adults (age 16 and older) living 
in households or prisons in the United 
States. 

• Relates the literacy skills of the 
nation’s adults to a variety of demo- 
graphic characteristics and explanatory 
variables and compares the results 
with those from the 1985 Young Adult 
Literacy Assessment and the 1990 
Workplace Literacy Survey. 

and older 


ed.gov/naal/ 


Beginning 

Teaching profession 

• Permits a better understanding of the 

Public school 

2007-08, 

http://nces. 

Teacher 

and job satisfaction, life 

impact of life events (such as getting 

teachers 

2008-09, 

ed.gov// 

Longitudinal 

events, and career mobil- 

married, moving to a new location, or 


2009-10, 

surveys/btls/ 

Study 

ity and movement to new 

starting a family) on teachers’ careers. 


2011-12 



schools 

• Provides insight into how school and 






district characteristics and policies 

affect teacher satisfaction and how 

teachers respond to transitions such 
as moving to a different school, 
changing grade level or subject taught, 
becoming a mentor, transitioning into a 
K-12 administrative position, or exiting 
the teaching field. 






• Contributes to policymakers’ under- 






standing of teachers and their careers 
as they enter, leave, or reenter the 
teaching workforce and make import- 
ant career and life decisions. 




Common Core of 

Public school universe 

• Provides basic statistical information 

Public 

Every year: 

http://nces. 


Data 


(for example, information 
about school location, 
enrollment, demographic 
characteristics); local 
education agency (phone 
numbers, number of high 
school graduates, type of 
agency); state aggregate 
nonfiscal data; state ag- 
gregate fiscal data; and 
district fiscal data 


on all children receiving a public edu- agencies: 
cation from prekindergarten through schools and 
grade 12 plus information on the government 

public funds collected and expended 
for public elementary and secondary 
education. 

Provides an official listing of public 
elementary and secondary schools 
and education agencies in the United 
States, which can be used to select 
samples for other National Center 
for Education Statistics surveys, and 
basic descriptive statistics on public 
elementary and secondary schools and 
schooling. 


1986- 

present 


ed.gov/ccd/ 
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1 Study 

Topics 

Purpose 

Respondents 

Years 

Survey link | 

High School & 
Beyond 

School, work, and home 
experiences; transitions 
to work; role in education 
of parents and peers; 

and education and occu- 
pational aspirations 

• Investigates the education and vo- 
cational development of high school 
students and how personal, social, 
and institutional factors shape this 
development. 

Students, 

parents, 

teachers, 

and school 

administrators 

1980, 1982, 
1984, 1986, 

1992 

http://nces. 

ed.gov/ 

surveys/hsb/ 

Education 
Longitudinal 
Study of 2002 

Social background, 
home education support 
system, school and 

classroom character- 
istics, postsecondary 

education choice and 
enrollment, employment, 

and outcomes 

• Monitors the transition of a national 
sample of young people as they prog- 
ress from grade 10 through high school 
and on to postsecondary education or 

careers. 

High school 
students, 
starting in 
grade 10 

2002, 2004, 
2006, 2012 

http://nces. 

ed.gov/ 

surveys/ 

els2002/ 

Fast Response 
Survey System 

Primary and secondary 
levels: racial/ethnic clas- 
sification; telecommuni- 
cations; libraries; edu- 

• Collects small amounts of data on key 
education issues within a relatively 
short timeframe (at all education 

levels from 1975 to 1990 and at the 

Local 

education 
agencies, 
public and 

1975- 

present 

http://nces. 

ed.gov/ 

surveys/ 

frss/ 


cation reform; violence 
and discipline problems; 
parental involvement; 
curriculum; nutrition edu- 
cation; teacher training/ 
professional develop- 
ment; school readiness; 
and principals’, admin- 
istrators’, and teachers’ 
perspectives 


elementary and secondary levels since 
1991). More than 80 surveys have 
been conducted on topics ranging 
from racial/ethnic classifications 
at the state and school levels; the 
availability and use of resources, such 
as advanced telecommunications and 
libraries; education reform; violence 
and discipline problems; parent involve- 
ment; curriculum placement and arts 
education; nutrition education; teacher 
training and professional development; 
vocational education; children’s readi- 
ness for school; and the perspectives 
of district superintendents, principals, 
and teachers on safe, disciplined, and 
drug-free schools. 


private 
schools, 
teachers, 
principals, and 
librarians 


High School 
Longitudinal 

Study of 2009 

Math and science educa- 
tion, changing environ- 
ment of high school, and 
postsecondary education 

• Focuses on understanding students’ 
trajectories from the beginning of high 
school into postsecondary education, 
the workforce, and beyond, including 
what students decide to pursue when, 
why, and how, especially but not solely 
in regard to science, technology, en- 
gineering, and math courses, majors, 
and careers. 

High school 
students, 
starting in 
grade 9; school 
administrators; 

school 

counselors; 
and parents 

2009, 2012, 
2013, 2016, 

2021 

http://nces. 

ed.gov/ 

surveys/ 

hsls09/ 

High School 

Student high school 

• Informs researchers and policymakers 

High school 

1982, 1987, 

http://nces. 

Transcript 

Studies Program 

records: courses, grades, 
class rank, and standard- 
ized scores 

about the course-taking patterns of 
students, which can then be analyzed 
in relation to students’ academic per- 
formance on assessment tests. 

students 

1990, 1992, 
1994, 1998, 
2000, 2004 

ed.gov/ 

surveys/hst/ 

National 

School, work, and home 

• Provides data on critical transitions 

Students, 

1988, 1990, 

http://nces. 

Education 

experiences; educational 

experienced by young people as they 

school 

1992, 1994, 

ed.gov/ 

Longitudinal 

Study of 1988 

resources and support; 
role in education of 
parents and peers; neigh- 
borhood characteristics; 
education and occupa- 
tional aspirations; and 
student perceptions 

leave elementary school and progress 
through high school into postsecondary 
institutions or the workforce and 

compares trends in the results of the 
National Longitudinal Study of the High 
School Class of 1972 and the High 
School & Beyond study. 

administrators, 
teachers, and 
parents 

2000 

surveys/ 

nels88/ 
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1 Study 

Topics 

Purpose 

Respondents 

Years Survey link 1 

National 

High school students 

• Provides information on the transi- 

High school 

1972, 1973, http://nces. 

Longitudinal 

and transitions into the 

tions of young adults from high school 

seniors 

1974, 1976, ed.gov/ 

Study of the High 

workforce 

through postsecondary education and 

(sampled again 

1979, 1986 surveys/ 

School Class of 

1972 


into the workplace. 

as adults) 

nls72/ 

Private School 

Private school type, 

• Builds an accurate and complete uni- 

Private schools 

Every two http://nces. 

Universe Survey 

length of school year and 

verse of private schools to serve as a 


years: ed.gov/ 


day, total enrollment, 

sampling frame for National Center for 


1989- surveys/pss/ 


demographics, number of 

Education Statistics surveys of private 


present 


graduates, and the num- 

schools. 




ber of teachers employed 

• Generate biennial data on the total 





number of private schools, teachers, 

and students. 



School Survey on 

School practices and pro- 

• Provides detailed information on 

School 

1999-2000, http://nces. 

Crime and Safety 

grams, parent and com- 

school crime and safety from the 

principals 

2003-04, ed.gov/ 


munity involvement at 

schools’ perspective. 


2005-06, surveys/ 


school, school security, 



2007-08, ssocs/ 


staff training, limitations 



2009-10, 


on crime prevention, 
frequency of crime and 
violence at school, the 
number of incidents, 
disciplinary problems 
and actions, and school 

characteristics 



2011-12 

Schools and 

Teacher demand, teacher 

• Collects the information necessary for 

Principals, 

1987-88, http://nces. 

Staffing Survey 

and principal character- 

a complete picture of U.S. elementary 

teachers, and 

1990-91, ed.gov/ 


istics, general conditions 

and secondary education. 

districts 

1993-94, surveys/ 


of the school, teacher 
compensation, district 
hiring and retention, 
teacher perceptions, and 

student characteristics 

• Provides national estimates of public 
elementary, secondary, and combined 
schools and teachers; state estimates 
of public elementary and secondary 
schools and teachers; and estimates 


1999-2000, sass/ 
2003-04, 

2007-08, 

2011-12 


of private schools and teachers at the 
national level and by private school 
affiliation. 


• Measures teacher, district, and school 
capacity (since 1999-2000), including 
teacher qualifications, teacher career 
paths, professional development, 
school organization and decision- 
making, curriculum and instruction, 
parent involvement, school safety and 
discipline, and school resources. 


Source: Burns, Wang, & Henning (2011); Thurgood, Walter, Carter, Henn, Huang, Nooter, et al. (2003). 
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Module 3: Writing survey items 


Overview 


Duration: 45-60 minutes, including 10-15 minutes for activity 3d 
Slides: Module 3 slide deck 
Handout: Handout 3d: Writing survey items 
Materials needed: 

• One copy of slides and handout 3d per participant. 

• Writing tools. 

• Projector and computer. 


Agenda 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 

• Activity 3.1. 


Background 


Whether adapting existing survey items for a new survey project or creating new items to 
cover questions and topics not found in existing sources, survey developers should follow 
best practices for writing survey items. This module describes considerations for writing 
concise survey items and effective response options and provides examples of effective 
survey formatting. The goal of this module is for participants to be able to identify and use 
best practices for writing survey items. 

Strategies for adapting module 3 to your context 


• If all participants are collaborating on the same project, this module can be used 
as a working session. When going through the slides, try to use examples from 
the participants’ own survey to discuss how the items are representative of the 
guidelines presented in this module, focusing on what might be improved in each 
item. Or go through the slide presentation first, present the activity, and then have 
participants write their own survey items or revise items on one of their existing 
surveys. 

• If all participants are not working on the same project, encourage them to bring 
or submit survey items that they are having trouble with or think are exemplary. 
Other participants can provide feedback about these items, either as a large group 
or in small groups. 
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Slides and notes for module 3 


Slide 1 


Slide 2 


Module 3 
presentation 

Writing 

items 


Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 



Slide 3 


Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 


Slide 4 


Many survey items are designed to gather information (for 
example, highest academic degree, age, gender, number of years 
teaching, and specific teaching practices). 

S To gather information. 


Purpose of survey items 
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Purpose of survey items 


Slide 5 


✓ 

✓ 


To gather information. 
To measure a construct. 


Some survey items are designed to measure a construct as 
accurately as possible (for example, job satisfaction, engagement, 
self-efficacy, and motivation). 


Slide 6 


Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 


Slide 7 


Goals for writing items 


Ensure validity. 


The overall goal is to ensure that survey items yield valid and 
reliable data that accurately represent what the researchers want 
to measure. 


Slide 8 


Goals for writing items 


Ensure validity. 

✓ Minimize item 
nonresponse. 


Minimize item nonresponse by ensuring that respondents are able 
and motivated to answer the item. Survey respondents may not 
be motivated to answer an item if it is unclear, irrelevant, or too 
difficult or if it asks for sensitive information. 
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Slide 9 


Goals for writing items 


v Ensure validity. 

/ Minimize item 
nonresponse. 

Minimize measurement 
error. 


Minimize measurement error by ensuring that: 

• Survey items are clear and concise so that all respondents will 
interpret them as intended. 

• All respondents can formulate an answer. 

• All respondents can figure out how to record their answers. 


Measurement error can come from several sources, such as 
respondents not answering items, giving inappropriate answers 
because they did not understand the item, or errors in recording 
answers. Every item should be written to minimize measurement 
error. To do so, ensure that instructions, items, and answer 
choices are simple, clear, and exhaustive. 

Measurement errors that are random or not systematic decrease 
reliability, meaning that the survey will produce less stable 
results over repeated administrations. Items with systematic 
measurement errors (that is, many responses are off in the same 
direction) produce less valid findings. 


Slide 11 


Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 



Slide 12 



The guidelines for writing items are: 

• Be specific. 

• Include definitions if necessary. 

• Avoid double-barreled items. 

• Avoid double negatives. 

• Use clear directions. 

• Use a reading level that is appropriate for the respondents. 
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Be specific 


Slide 13 


• Do you work full-time? 

• About how many total hours per 
week do you usually work for 
pay, counting all jobs? 


Provide enough details for consistent interpretation of an item. For 
example, different people could interpret “Do you work full-time?” 
differently. It would be better to ask: “About how many total hours 
per week do you usually work for pay, counting all jobs?” 

In addition to being more consistently interpreted, asking the item 
this way allows for multiple options for analyzing the answers 
(for example, looking at those who work more than 30, 35, or 40 
hours per week). 


Slide 14 


Slide 15 


Include definitions if necessary 






l XT3 

& oi 
n 

c tcA 








# D.t r->: 

„ , . ch® ' 
\ocH 
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Each item should represent a unidimensional 
concept. 


If you are not sure that all respondents will interpret a term as 
you intend, provide a definition. For example, to ensure that all 
principals interpreted the terms coaches and mentors in the same 
way on a professional development survey, these definitions 
should be included on the survey. 

• A coach is someone who supports the principal’s professional 
development by providing structured and consistent leadership 
focused on achieving a specific set of performance goals. 

• A mentor is someone who provides informal advice and 
information to a principal on a variety of issues, which are not 
necessarily focused on achieving a specific goal. 

Each item should represent one concept. An example of a double- 
barreled item is: “To what extent are your measures of progress 
quantitative and objective?” Ask two separate items: 

• “To what extent are your measures of progress quantitative?” 

• “To what extent are your measures of progress objective?” 


Slide 16 


Avoid potential double 
negatives 


It is more straightforward to phrase this item positively: “My 
child’s teacher cares about my child very much.” 


• My child's teacher does not seem 
to care much about my child. 

Strongly agree/ Agree/ Disagree/Strongly disagree 


• My child's teacher cares about 
my child very much. 

Strongly agree/ Agree/ Disagree/Strongly disagree 
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Use clear directions where 
necessary 


• Select all that apply. 

• Select only one response. 

• Round to the nearest whole 
number. 

• Do not include long-term 
substitute teachers in your total 
count. 


Slide 18 


Slide 19 


Slide 20 



For adult surveys, a general rule of thumb is to use language at 
the grade 8 reading level. Avoid technical jargon that may not be 
understood by everyone. Provide definitions for key terms when 
necessary. 


Closed-ended items ask the respondent to pick from one of 
several options. 

Open-ended items allow respondents to provide their own 
responses. These can be a short answer, a fill in the blank, or a 
long answer. 

Plan for how these answers will be categorized, analyzed, and 
reported. Analyzing and summarizing open-ended items can be 
very time consuming. 


What are the advantages of closed-ended items? 

• More reliable responses. 

• Much quicker to analyze. 

• Easier to interpret. 

• Can incorporate an “Other, please specify” option to capture 
information missing from closed-ended options. 
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Advantages of 
open-ended items 


Slide 21 


Slide 22 


What are the advantages of open-ended items? 

• Unanticipated answers can be obtained (desirable if the range 
of responses for a closed-ended item is unknown). For example, 
you may want to ask respondents “What aspect of professional 
development do you find most useful?” rather than providing a 
list of professional development features that may or may not 
be comprehensive. 

• Respondents may prefer to answer questions in their 
own words. For example, if you want to better understand 
respondents’ career pathways, you may ask them to describe 
their trajectory in their own words in lieu of answering several 
closed-ended items. 

• Open-ended answers on a field test can provide closed-ended 
options for a main study. For example, a field test of a survey 
on mentoring models may yield a finite set of mentoring models 
used in a school (such as coaching, peer-to-peer, formal 
mentor, and informal mentor) that can then be used in closed- 
ended items. 


Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 
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Types of rating scales 


• Agreement. 

• Interest. 

• Importance. 

• Frequency. 

• Degree or extent. 

• Similarity ( like me to not like me). 


Survey respondents have a tendency to agree with statements 
when given the usual strongly agree to strongly disagree scale, 
so alternatives should be considered when appropriate, such as 
not very interested to very interested, not very important to very 
important, never to very often, none to a lot, and very dissimilar to 
very similar. 


Slide 24 


Types of rating scales 


• Unipolar scale: ranges from lack 
of attribute to a high level of the 
attribute. 
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Unipolar example 


Slide 25 


54. How much actual control do you have M YOUR CLASSROOM at thia school over the folowaig 
areas of your planning and teaching? 


* Selecting textbooks and^ 

£ Marti (X) one b 


No 

control 

Minor 

Moderate 

A great deal 
of control 

. □ 

> □ 

. n 

. □ 

b. Selecting content, topics. 

and skills to be taught oar 

- □ 

a n 

. n 

* □ 

c. Selecting teaching ox 

- □ 

. n 

. n 

. □ 

d. Evaluating and grading ^ 

- □ 

a □ 

a n 

a □ 


Slide 26 


Types of rating scales 


• Unipolar scale: ranges from lack 
of attribute to a high level of the 
attribute. 

• Bipolar scale: ranges from 
positive to negative (for 
example, agree to disagree ). 
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Bipolar example 


63. To what extent do you agree or disagree with each of the following statements? 


4 Mark (X) one box on each line 


a. The school administration's behavior 

toward the staff is supportive and m 

encouraging. 

W 

Somewhat 

agree 

S d°i^e h f 


t □ 

2 □ 

3 ° 


b. 1 am satisfied with my teaching salary. oos 

*. □ 

2 □ 

3 D 

4 □ 

c. The level of student misbehavior in this 
school (such as noise, horseplay or 
fighting in the halls, cafeteria, or student 
lounge) interferes with my teaching. 

,i □ 

2 □ 

3 □ 

4 □ 

d. 1 receive a great deal of support 
from parents for the work 1 do. 

t □ 

2 □ 

3 1=1 

4 □ 
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Types of rating scales 


• Unipolar scale: ranges from lack 
of attribute to a high level of the 
attribute. 

• Bipolar scale: ranges from 
positive to negative (for 
example, agree to disagree ). 

• Match the type of scale to the 
structure of the attribute. 
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Types of rating scales 


• Unipolar scale: ranges from lack 
of attribute to a high level of the 
attribute. 

• Bipolar scale: ranges from 
positive to negative (for 
example, agree to disagree ). 

• Match the type of scale to the 
structure of the attribute. 
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Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 


Writing response options 
for scales 


• Four to six response categories 
typically are used. 
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Writing response options 
for scales 


• Four to six response categories 
typically are used. 

• Use middle (undecided or 
neutral) categories thoughtfully. 


My teacher grades fairly. 

• Strongly disagree. 

• Disagree. 

• Neutral. 

• Agree. 

• Strongly agree. 
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There are no rules about offering a neutral category. 

In some instances, it might be reasonable. However, respondents 
often choose the middle category because of a lack of knowledge, 
uncooperativeness, difficulty in reading or understanding, 
reluctance to answer, or inapplicability. 

Think carefully about what makes the most sense given the 
context. 

You can also offer a not applicable or don't know response option 
if these are legitimate answers. These options should be off to 
one side, not in the middle. 

For example, to measure attitudes toward a professional 
development program, you may allow respondents to choose not 
applicable if they did not participate in the activity so as not to 
force inaccurate or missing responses. 

In general, label all response options. Respondents will assign 
their own (possibly inconsistent) meanings to numbers that are 
not labeled. 


1 2 3 4 5 

Rarely Frequently 
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Writing response options 


Do not use rating scales with 
unlabeled options: 
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Response options example 
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Make responses 
mutually exclusive 


• What is wrong with this item? 


Response options must be mutually exclusive. There must be only 
one possible correct answer. 

A better set of responses for this question is as follows: 

• 0-4 years. 


How many years have you been • 5-9 years, 

teaching? 

. o-5 years. • 10 or more years. 

• 5-10 years. 

• 10+ years. 
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Make responses 
collectively exhaustive 


• What is wrong with this item? 


Response options must be collectively exhaustive. 

What if a teacher never assigns homework? There should be an 
option for never. 


How often do you do assign homework? 

• Monthly. 

• Weekly. 

• Daily. 
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Frequency scale example 


25. To the best of your knowledge, how often do the following types of problems occur at 


Physical conflicts among students 0,49 



Ha ST 


s 

Hat sr 


, □ 

2 □ 

2 □ 

. □ 

• □ 

Robbery or theft 0150 

i □ 

« n 

3 □ 

. n 

. □ 

Vandalism 

, □ 

2 □ 

2 □ 

. □ 

I □ 

Student use of alcohol 

, □ 

2 □ 

2 □ 

4 □ 

. □ 

Student use of illegal drugs 0183 

. □ 

2 □ 

2 □ 

. □ 

. □ 

Student possession of weapons 0154 

, □ 

2 □ 

2 □ 

4 □ 

. □ 

Physical abuse of teachers 0186 

, □ 

2 □ 

2 □ 

4 □ 

. □ 

Student racial tensions 0180 

- □ 

2 □ 

2 □ 

4 □ 

2 □ 
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Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 
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Having fewer items per page is better than crowding more items 
onto fewer pages. The survey will appear less burdensome if it 
has plenty of white space and an uncrowded layout. 

Respondents may be put off by seeing sensitive questions, such 
as questions about the respondent’s race/ethnicity and income, 
at the beginning of the survey and may therefore be less likely to 
complete the survey. Placing sensitive items at the end allows 
the respondent to become engaged in the survey content before 
encountering them. 
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Slide 41 


Formatting considerations 


• Order all rating scales on the 
survey in one direction 



(either negative to positive or 
positive to negative). 
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Shading alternate rows helps the respondent read the items more 
easily and minimizes error. This strategy works for both paper and 
online surveys. 

• Shade alternate rows. 


Formatting considerations 




Pull-out (i.e„ out of < 



□ 

□ 

□ 

□ 

□ 

□ 


No 

□ 

□ 

□ 

□ 

□ 

□ 
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Outline 


• Purpose of survey items. 

• Goals for writing items. 

• Guidelines for writing items. 

• Rating scales. 

• Response options. 

• Formatting considerations. 

• Skip patterns. 
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Clarify skip instructions 



' 0 aSESSpS* * 3 - 

i □ Standards-based evaluation (Evaluation is 
by district or school policy-makers as being 
essential to effective teaching.) 


Sometimes survey developers do not want every survey 
respondent to answer every item. They use skip patterns, which 
direct some respondents to skip items that are not relevant to 
them. 

Instructions for skip patterns must be very clear. The illustrated 
survey item for principals is complex but clear. However, this level 
of complexity would not be appropriate for a student survey. 

Some tips for skip instructions: 

• Make it clear when an item should be skipped. 


• Use arrows as visual clues in paper-and-pencil surveys. 


• Avoid overly complicated skip instructions for paper-and-pencil 
surveys. 


• In online surveys, skips can be automated, which makes 
navigating a survey with many skips appear seamless to the 
respondent. 


43 










Slide 45 


Additional resources 


Converse, J. M v & Presser, S. (1986). Survey questions: Handcrafting 
the standardized questionnaire. Newbury Park, CA: SAGE. 

Fowler, F. J., Jr. (2009). Survey research methods. In L. Brickman & 

D. J. Rog (Series Eds.), Applied social research methods series: 
Voi 1. Newbury Park, CA: SAGE. 

Lavrakas, P. J. (2008). Encyclopedia of survey research methods. 
Thousand Oaks, CA: SAGE. 

Swanlund, A., Brown, M., & Martin, B. (2012). HSSI Survey 
Workshop II. Chicago, IL: American Institutes for Research. 
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Additional resources 


For more information, please visit the following websites: 

• The American Association for Public Opinion Research: 
http://www.aapor.org/ . 

• Public Opinion Quarterly: https://poa.oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.ora/content/current . 
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Activity 3.1: Writing survey items 


This 10-15 minute activity in handout 3d asks participants to improve survey items using 
best practices. It is best administered after the slide presentation. If the group of partic- 
ipants is small, encourage them to work on the items individually for approximately 5 
minutes before convening the whole group and asking volunteers to share their improved 
items. If there are many participants, encourage them to work in small groups before 
reporting out as a whole. Leave time to review the answers so that participants can develop 
a sense of whether they understand the material in this module. 

Consider using the suggestions given below each of the following items from handout 3.1 
to show how they can be improved: 

1. How do you feel about your job as a principal? Select one. 

a. Fulfilling 

b. Fulfilling but stressful 

c. Not fulfilling and very stressful 

Problem: The response options are not mutually exclusive or collectively exhaustive. 

Revision: Use two separate items with rating-scale response options, such as: “How stressful 
is your job as principal?” with a rating scale from very stressful to not stressful and “How 
fulfilling is your job as principal?” with a rating scale from very fulfilling to not fulfilling. 

2. How many days was your child late to school? Select one. 

a. 1-5 

b. 6-10 

c. 10 or more 

Problem: The response options are not mutually exclusive (two categories include 10) and 
are not exhaustive (no zero). In addition, no timeframe is mentioned, so a parent will not 
be sure how far back to count. Some parents might also wonder what is considered late. 

Revision: An improved item solves each problem: “In the past month, how many days was 
your child late for the start of the school day by at least 15 minutes?” with response options 
of 0, 1-2, 3-5, 6-10, and more than 10. 

3. How often do students in your class use a computer to write? Select one. 

a. Never 

b. Sometimes 

c. Often 

d. Frequently 

Problems: “Use a computer to write” is vague. No indication is given of whether the writing 
must occur in class or can occur at home. No timeframe is given. The response options are 
also vague. One respondent might think that using a computer to write twice a month 
qualifies as “often,” whereas another might think that using a computer to write twice a 
month warrants a “sometimes” response. 
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Revision: Each of these problems is solved in the following revision: “Since the start of the 
school year, how frequently did students in this class use a computer during class time to 
work on writing assignments?” with response options of never, less than once per month, 
about once per month, a few times per month, about once per week, and more than once 
per week. 

4. Do you agree that the principal has a strong leadership style and is respected for 
it? Select one. 

a. Yes 

b. No 

c. Maybe 

Problem: This survey item is double barreled; it asks the respondent to answer whether the 
principal has strong leadership and whether he or she is respected for such leadership. A 
respondent may agree that the principal demonstrates strong leadership but disagree that 
he or she is respected for it. Also, the option “maybe” is not a reasonable response for this 
kind of item. 

Revision: This item can be split it into two separate items — one asking about the level of 
leadership (with a strongly agree to strongly disagree or very weak to very strong rating scale) 
and one asking about the principal being respected (with a strongly agree to strongly disagree 
or not respected to highly respected rating scale). 

5. How many times have you visited your child’s classroom? 

Problem: This survey item is vague and parents may interpret it in different ways. 

Revision: The specificity of the item can be improved with more details about what a “visit” 
means and the addition of a timeframe. For example, “Since the beginning of this school 
year, how many times have you visited your child’s classroom during the school day to 
observe the class or participate in an activity?” 
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Handout 3.1: Writing survey items 


What problems do you see with these survey items, and how can they be improved? 


L How do you feel about your job as a principal? Select one. 

□ Fulfilling 

□ Fulfilling but stressful 

□ Not fulfilling and very stressful 

Problem: 

Revision: 


2. How many days was your child late to school? Select one. 

□ 1-5 

□ 6-10 

□ 10 or more 

Problem: 

Revision: 


3. How often do students in your class use a computer to write? Select one. 

□ Never 

□ Sometimes 

□ Often 

□ Frequently 

Problem: 

Revision: 


4. Do you agree that the principal has a strong leadership style and is respected for 
it? Select one. 

□ Yes 

□ No 

□ Maybe 

Problem: 

Revision: 


5. How many times have you visited your child’s classroom? 

Problem: 

Revision: 
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6, Write a few survey items to address one of the research questions you identified in 
activity hh 
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Module 4: Pretesting survey items 


Overview 


Duration: 60 minutes, including 30 minutes for activity 4.1 
Duration of activity : 30 minutes 
Slides: Module 4 slide deck 

Handout: Handout 4.1: Conducting cognitive interviews 
Materials needed: 

• One copy of slides and handout 4.1 per participant. 

• Writing tools. 

• Projector and computer. 

Agenda 

• Methods to test new survey items, 
o Expert reviews. 

o Cognitive interviews, 
o Focus groups, 
o Field testing. 

• Activity 4.1. 


Background 


Pretesting, which evaluates the items intended for the final survey, can occur in one of four ways: expert review, 
cognitive interviews, focus groups, or field testing. This module focuses on cognitive interviews as a tool for pre- 
testing items because it is the most common method, and the interviews allow for the most detailed feedback 
regarding new or revised survey items. Ideally, if time and resources allow, a project will use all four methods 
to some degree. Depending on the nature of the survey and available resources, survey developers may have to 
choose the most appropriate method. The goal of this module is for participants to understand the importance 
of pretesting items and the methods used for pretesting. 

Strategies for adapting module 4 to your context 


• If you are using this module as a working session, consider using the presentation as a precursor to a 
working session in which the team writes a cognitive interview protocol and introduction appropriate 
for their survey project. 

• If a team has already developed a survey and a cognitive interview protocol and intends to use this 
module to train staff unfamiliar in cognitive interviewing techniques, then use this presentation and 
substitute its survey and protocol for activity 4.1. 
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Slides and notes for module 4 


Slide 1 


Slide 2 



Module 4 
presentation 

Pretesting 
survey items 



Pretesting is a set of processes for reviewing items before using 
them in the final survey (Lavrakas, 2008). Pretesting increases 
the validity of a survey by ensuring that the relevant aspects of the 
topics are included and that respondents understand the items as 
intended. 

The results of pretesting will inform revisions to the survey to 
improve its validity. 

If items have been administered or tested previously but will be 
used in a different context, with a different type of respondent, or 
in a different survey mode, you should still pretest them to ensure 
that they will work well in your context. 


Slide 3 




Set pretest 
goals. 


Pretesting should have its own set of specific goals. What do you 

want to learn about? Example goals could include: 

• To ensure that the survey items appropriately capture all facets 
of the topic under investigation. 

• To examine whether respondents understand what the survey 
items are asking. 

• To test response categories so that all respondents interpret 
them in a similar way. 


Depending on the goals of pretesting, survey developers may 
employ one, or possibly all four, pretesting methods. 

Methods to test new survey items: 

• Expert reviews. 

• Cognitive interviews. 

• Focus groups. 

• Field testing. 


Outline 
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Slide 5 


Outline 


Methods to test new survey items: 

• Expert reviews. 

• Cognitive interviews. 

• Focus groups. 

• Field testing. 
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If you ask experts to review a draft of the survey, give specific 
guidance about what aspects of the survey you want them 
to focus on. Experts may provide valuable insights into what 
important items might be missing from the survey or how to 
improve the wording of survey items given their knowledge of 
certain respondent types. Be sure to provide background about 
the purpose and eventual use of the survey results so that the 
experts can offer the most appropriate feedback. 

Expert reviews can also occur at other points in the survey 
development process. For example, you may ask experts to 
suggest surveys from which to borrow items, review findings from 
other pretest methods, or help make final revisions. 



Experts can be academics, practitioners, or other professionals 
who are knowledgeable about the topics covered in the survey. 
Who are the leading experts in the topic you want to study? 

Although expert reviewers can provide depth of knowledge 
on a subject, they often do not represent the viewpoint of 
the respondent. Therefore, pretesting surveys with potential 
respondents is also encouraged. 
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Outline 


Methods to test new survey items: 

• Expert reviews. 

• Cognitive interviews. 

• Focus groups. 

• Field testing. 
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Slide 9 
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A cognitive interview is a one-on-one interview between an 
interviewer and a respondent that is designed to reveal how 
respondents understand and respond to the survey items. The 
technique draws from the field of cognitive aspects of survey 
methodology (Jabine, Straf, Tanur, & Tourangeau, 1984; Lavrakas, 
2008). 

Four key processes are investigated: 

• How respondents understand and interpret an item. 

• How respondents retrieve information to answer the item. 

• The process respondents use for making a decision. 

• How respondents answer the item. 

A cognitive interview is an appraisal of the items being 
administered. It is geared toward uncovering sources of error. 


Cognitive interviews: Purpose 


A cognitive 
interview is 
an appraisal 
of the survey 
questions. 
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Cognitive interviews: Purpose 


A cognitive 
interview is 
not a test. 



A cognitive interview is not a test of whether the respondent 
answers the items correctly. Although the respondent’s answer 
is of interest, specifically in identifying response errors, the 
interviewer is equally interested in how the respondent arrives at 
that answer, how the respondent interacts with the items, and 
what the respondent says. 
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Cognitive interviews: Schedule 


This slide shows a typical cognitive interview schedule, beginning 
with documentation of the research questions for the cognitive 
interviews and ending with a final summary of interview findings 
and recommendations for the final survey draft. 



Round 1 usually catches major misunderstandings of the items, 
particularly if they are new. After round 1, review your notes and 
revise the items as necessary. 

Ideally, you will have time to conduct at least one more round to 
ensure that the revised items eliminate any confusion identified in 
round 1. 


Multiple rounds with multiple respondents (for example, 
three rounds with nine respondents each) may be needed to 
incorporate feedback, revise items, and test the revised items. 
As a general rule of thumb, cognitive interviews can cease when 
no new sources of error are discovered after interviewing several 
respondents. 


You may wish to write a final report that details the number of 
interviews conducted, important respondent characteristics, and a 
summary of your procedure and findings. 
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Slide 13 



The basic outline of a cognitive interview is: 

• Recruit individuals from the target population. 

• Identify who will conduct the interviews. 


Structure 

your 

cognitive 

interview. 



• Train interviewers on the cognitive interviewing techniques 
discussed in this module. 

• Conduct the interviews. 

• Compile and review interview notes. 
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It is important to make sure that the respondents who participate 
in cognitive interviews are similar to the respondents who will 
take the survey. For example, if you are designing a survey about 
teacher professional development and perceptions about the 
usefulness of training content, think about all kinds of teachers 
who will participate in the final survey (for example, new teachers 
and science teachers). 

Recruit cognitive interview participants who are similar to the 
sample (and important subgroups) who will take the final survey. 

• Recruitment is similar to that of focus groups (see module 8). 

• Use flyers, newspapers, the Internet, and points of contact for 
rare populations. 

Who should do the interviews? 

Someone: 

• With good interpersonal skills. 

• With knowledge of issues related to measurement. 

• Familiar with surveys and survey design. 

• Familiar with the survey topic. 

(Willis, 2005) 
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Interview approaches 


It is not the interviewer’s role to explain what an item means; 
rather, the interviewer gets the respondent to talk about what he 
or she is or was thinking when answering the item (that is, what 
the respondent thinks the item is asking). 


• Concurrent "think-aloud" 

• Retrospective probing 


Cognitive interviewers often employ two types of interview 
approaches: concurrent think-alouds and retrospective probing. 
The goal of the interview will determine which approach is used. 
Interviewers can use either method or a mixture of both. 
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What happens in concurrent think-alouds? 

• Respondents verbally tell you what they are thinking as they 
answer an item. 

• Respondents allow the interviewer to understand their cognitive 
processes in answering the item. 

In a concurrent think-aloud, the interviewer asks the respondent 
to read the item aloud and verbalize aloud what he or she is 
thinking about when formulating an answer. After the respondent 
thinks through and gives a response, the interviewer should seek 
clarification on statements that may represent sources of error. 
Concurrent think-alouds are useful when knowing the process that 
a respondent uses to answer an item is important. 

For example, suppose that you are testing an item that 
asks parents about weekly, regularly scheduled child care 
arrangements with relatives (“Does this child have any regularly 
scheduled relative care arrangements in a typical week?”). To 
answer this item, parents are required to think through who cares 
for the child, and whether the care is regularly scheduled on a 
weekly basis. 

The interviewer should pay attention to how the parent answers. 
Parents may forget about certain days or times of the week, such 
as weekends or evenings, when a child is cared for by a relative. 
Are all days of the week mentioned, or are there gaps in days or 
times (are evenings mentioned?). If, after several interviews, it is 
determined that parents are omitting relative care arrangements, 
the item may need to be revised, either through changes in 
wording or format. 

Concurrent think-alouds also may be useful when you are testing 
a brand new item that has never been used before. Sometimes 
important sources of error can be found by just having the 
respondent think aloud while answering an item. 

In retrospective probing, interview questions are asked after 
the respondent has completed a task: an item, a section of the 
survey, or the entire survey. Interview teams should determine 
which is most appropriate for their needs (Willis, 1999). 

Retrospective probing provides targeted information about how 
respondents navigate through, interact with, and answer the 
items. Retrospective probes are useful when testing items that 
have been used before, or in later stages of survey development 
when finalizing the survey and ensuring that no lingering issues 
with understanding remain. 

Retrospective probing is less taxing for the respondent than 
concurrent think-alouds, but some detail in how respondents 
answer an item may be lost because respondents may be unable 
to accurately recall what they were thinking when they answered 
an item. 
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ink-aloud" 
advantages 
and 

disadvantages 



What are the advantages of think-alouds? 

• There is less concern for interviewers leading respondents and 
causing bias. 

• Less interviewer training is involved because respondents do 
most of the talking while the interviewer actively listens. 

• The open-ended format lends itself to richer information that 
otherwise may not be collected. 

What are the disadvantages of think-alouds? 

• Respondents must be given an example of a probe because not 
everyone is used to thinking aloud. 

• The burden is on respondents because they are required to 
verbalize their thoughts while interviewers actively listen and jot 
down notes. 

• Some respondents may not explain their thought process and 
just give an answer to the survey item. 

• Some respondents may stray from the survey item and 
interview question at hand. 

(Willis, 1999) 

What are the advantages of retrospective probing? 

• It is less taxing for respondents because they are not required 
to verbalize their thoughts. 

• Retrospective probes are a more structured type of interview, 
where the interviewer asks predetermined interview questions 
and the respondents answer. Therefore it can be easier to keep 
respondents focused on the survey item and interview question 
at hand. 

• The interviewer has more control of the interview (compared 
with concurrent think-alouds) and can ensure that the 
respondent stays on task. 

What are the disadvantages of retrospective probing? 

• The more structured interview protocol (compared with the 
more open-ended think-aloud) could lead to respondents 
giving only superficial answers because many of the interview 
questions are predetermined. 

• The interviewer could unintentionally — by the way that 
questions are phrased or asked — lead respondents to answer 
in a particular way. 

• More interviewer training may be needed on the cognitive 
interview protocol and follow-up interview questions (for 
example, scripted and spontaneous probes) based on feedback 
from respondents. 
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Types of probes 


Slide 21 


Slide 22 


Scripted 

and 

spontaneous 


Scripted probes are predetermined interview questions that 
all respondents are asked. Scripted probes are based on the 
cognitive interview protocols and driven by the goals of the 
interviews. 

Spontaneous probes are interview questions that arise out 
of something specific that the respondent said. Allowing the 
interviewer to react to what a respondent says may uncover 
sources of error that were previously unknown. 

In practice, most cognitive interviews include a combination of 
both scripted and spontaneous probes (in other words, interview 
questions; Willis, 1999). 

Retrospective probes are a type of scripted probe that are asked 
of everyone and are driven by the goals of the interviews (for 
example, what you want to know). However, sometimes more 
spontaneous probes are needed to further clarify what the 
respondent says. The following slides provide examples of probes 
that interviewers can use and adapt. 


Cognitive interview 
example probes 


• Comprehension or interpretation: What does the word 
equivalent mean to you? 

• Paraphrasing: In your own words, what is this item asking you 
to do? 



• Confidence judgment: Can you tell me how sure you are that 
you...? 

(Willis, 1999) 
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Types of probes 


• Recall: How do you remember...? 

• Specific: Why do you think this item was asked? 

• General: Was it easy or hard to answer this item? 
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Possible probes: 

• Were there any items you were unsure about? 

• Does this cover all of your involvement? 

Unscripted follow-up probes: 

• Probe: Was this item easy or hard for you to answer? 

• Respondent: It was easy! 

• Follow-up probe: Can you tell me why it was easy? 
(Willis, 1999) 
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As respondents arrive for the cognitive interviews, have them 
complete any necessary documentation to participate, such as 
signing an informed consent form. If interviews are recorded, this 
must be included in the informed consent form and made known 
to the respondent. 

In the interview process, the interviewer: 

• Begins with an overview of the purpose of the study. 

• Demonstrates an example of a “think-aloud” probe (if this 
approach is used) so that the respondent is clear about what is 
expected. 

• Asks questions of interest based on the interview protocol. 

• Ends the interview. 

Cognitive interviews usually take 30-60 minutes. (The interviewer 
should be cognizant of the time. If the survey is long, the 
interviewer should prioritize sections and items for which more 
information is needed.) 

After completing the cognitive interview, review the notes (and 
audio tapes if sessions were recorded) as soon as possible so 
that important details are not forgotten. 
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Cognitive interviews: Analysis 


Review notes 
and listen to 
audio 

recordings of 
interviews. 


Analysis is done by reviewing notes and listening to audio 
recordings of interviews: 

• Look for common themes across interviews. 

• Return to the research questions and discuss findings and 
unexpected problems. 

• Revise problematic items to improve their clarity. 

• Discuss items with the team, make changes, and conduct more 
interviews. 

• Note that the analysis may take twice as long as the interviews. 

• Use later interviews to fine-tune the survey. 
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Cognitive interviews: Example 1 




For the purposes of the questions below, please consider the following definitions: 







Respondent 2 had problems with the definitions. Suggested 
percentages in definitions to clarify the categories are as follows: 

• Slight oversupply = 25 percent higher than what I needed. 

• Extreme oversupply = 50 percent higher than what I needed. 
How could you modify the survey item based on this feedback? 

The response categories should be clearly defined so that all 
respondents interpret them in a similar way. Cognitive interviewers 
asked respondents to tell, in their own words, what the words 
shortage and oversupply meant, but the respondents went 
through different thought processes to answer the item. After the 
interviews, the team revisited the item’s focus. 
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Cognitive interviews: Example 1 



The cognitive interview team decided that the item was really 
trying to capture how easy or difficult it was for the district to hire 
teachers in specific areas. 

Rather than asking respondents to differentiate between extreme 
shortage and extreme oversupply, they turned to response 
categories borrowed from a National Center for Education 
Statistics survey. This slide shows the revised item. 
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Cognitive interviews: Example 2 


Substitute Teachers 

7. How much difficulty did your district have during the 2010-11 and 2011-12 academic years in 
securing substitute teachers? 


The respondent stated that there was a difference between 
finding short-term and long-term substitutes. How would you revise 
the survey item based on this feedback? 


A lot of difficulty 
Some difficulty 
A little difficulty 
No difficulty 


O 
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Cognitive interviews: Example 2 


8. How much difficulty did your district have during the 2011-12 or 2012-13 academic years in 



Easy Somewhat Very 

difficult difficult 

Short-term substitute 

o : o o 

Long-term substitute (>15 days) 

'“'o' i 6 I o' 


9. How much difficulty do you anticipate having to secure substitute teachers in the next five 
years ? Mark one response for each row. 



Easy : Somewhat Very 

difficult difficult 

Short-term substitute 

O | O O 

Long-term substitute (>15 days) 

6:0 
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Outline 


Methods to test new survey items: 

• Expert reviews. 

• Cognitive interviews. 

• Focus groups. 

• Field testing. 
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Ask 

participants 
to comment 
on survey 
items as a 
group. 


Focus groups are a type of qualitative research that can be 
used to pretest items during survey development. Focus groups 
typically have one or two moderators and six to eight participants. 
Depending on the goal of the pretest, the participants may include 
experts or potential respondents. 

Focus groups can provide a better understanding of whether the 
survey covers all the important elements of the topic and how 
typical respondents interpret or react to the items on the survey. 
The group format allows for discussions about how well the items 
cover all the relevant aspects of the survey topic. Focus groups 
may highlight potential sources of error, but because of the group 
setting it may be difficult to get into detailed discussions about 
that error. Focus groups typically last 60-90 minutes. Additional 
information on focus groups is in module 8. 

Focus group participants can be asked to comment on an early 
draft of a survey as a group to: 

• Identify unclear concepts. 

• Explain how they conceptualize each topic on a list. 

Generally, focus groups are best suited for generating ideas about 
an unfamiliar topic. They can also inform researchers on how to 
write items to capture accurate responses. 

Cognitive interviews are the preferred method when you have a 
good understanding of the topic you are trying to measure and 
want to refine the survey items. 
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Outline 


Methods to test new survey items: 

• Expert reviews. 

• Cognitive interviews. 

• Focus groups. 

• Field testing. 
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A field test is a small-scale administration of the survey designed 
to evaluate the survey items and administration procedures. 

Field test data may suggest item revisions if, for example, there 
are many missing responses for certain items or nearly all the 
respondents answered an item with the same response. Typically, 
high-stakes surveys include a field test. 

Field tests can include methodological experiments (for example, 
comparing approaches to maximizing response rates or alternative 
item wording). 

The sample size for a field test is determined by the budget 
available. Typically, the larger the sample, the more comparisons 
you can make between different groups. Field-test samples should 
reflect the characteristics of the target population. 
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Activity 3: Conducting 
cognitive interviews 
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Additional resources 


For more information, please visit the following websites: 

• The American Association for Public Opinion Research: 
http://www.aapor.org /. 

• Public Opinion Quarterly: https://poa. oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.org/content/current . 
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Activity 4.1: Conducting cognitive interviews 


This activity in handout 4.1 contains two exercises in which participants can take turns 
being the cognitive interviewer and the respondent. The individual playing the role of 
the cognitive interviewer should use the protocol in exercise 1 of handout 4d to probe the 
respondent, who will fill out the short survey in exercise 1 of handout 4.1. After the first 
exercise, the pair should switch roles — the respondent becomes the interviewer, and the 
interviewer becomes the respondent — to complete exercise 2. Each exercise should take 
about 15 minutes. The following schedule should provide sufficient time for participants to 
get a sense of the purpose and general structure of a cognitive interview: 

• Introduce activity and distribute handout 4d: 2-3 minutes. 

• Exercise 1: About 15 minutes. 

• Exercise 2: About 15 minutes. 

• Reconvene the whole group (if time allows): What was it like for the participants 
to “think out loud”? To reflect on their thought process? What else did participants 
notice? 

If there is insufficient time to complete both activities, consider having participants work 
through exercise 1 and either reserve exercise 2 for later or work through it during lunch 
or a break. 
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Handout 4.1: Conducting cognitive interviews 


Directions 

Break into pairs. During exercise 1, one person completes the sample survey, and the other 
person is the cognitive interviewer. The cognitive interviewer should read all the probes 
based on the interview protocol for exercise 1 and take good notes on what the respondent 
says. The interviewee should answer the questions as if participating in a real cognitive 
interview. After the interview, the two participants switch roles and complete exercise 2. 
Each exercise includes an open-ended item to think through out loud as well as several 
multiple choice items. 
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Exercise 1 


For cognitive interviewer : Protocol for conducting a cognitive interview for exercise 1 
( Note to interviewer: This script should he read to the respondent .) 

I would like to begin by thanking you for coming here today to help us. We’re scheduled 
to be here for 15 minutes, and I will make sure that we do not go beyond that time. We’ve 
asked you to come because your local school district is conducting a survey on residents’ 
attitudes about various education issues. Today, I’m going to ask you to fill out a survey and 
provide feedback so that we can help the school district make sure that all different types 
of residents will understand the actual survey. 

To help keep track of everything you say today, I’m going to audio record this session. I’ll 
also be taking notes, so you might notice me writing as we are talking. I want you to know 
that your responses are voluntary and confidential. 

In just a minute I’m going to hand you a short survey, and I’d like you to fill it out the same 
way you would if it came to you at home in the mail. Unlike the case of responding to a 
survey at home, though, I’ll ask you to stop periodically so that we can talk about what you 
were thinking as you answered the items. 

There also will be one item for which I will ask you to think “out loud.” For this item, it is 
important that you tell me what you are thinking and feeling while you are actually in the 
process of coming up with your answer. 

Now, because some people aren’t used to reading, thinking, and expressing their feelings 
aloud, I’d like to begin today with a short practice item. Please read out loud the item I’m 
about to give you and tell me your thoughts and feelings from the moment I handed you 
the item until you finish explaining your answer. 

(Note to interviewer: Give the practice item on the next page to the respondent for the think - 
aloud exercise .) 
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Question: How many times did you eat a meal at a restaurant in the past week? 


64 


Thank you. I’m going to hand you the sample survey now. Please read it and fill it out as if 
you were responding to the actual survey. As you go through the survey, if any item sticks 
out to you in any way (maybe you think an item is hard to answer, difficult to understand, 
located in a strange place, or you just don’t like it) — anything at all that you would like to 
say about an item — please tell me so that we can discuss it. 

(Note to interviewer: Give survey to respondent.) 


65 


For respondent: Survey for exercise 1 


This survey on education issues will be mailed to residents of a local school district. We are 
testing the items for comprehension and clarity. 

7. How much confidence do you have in your local schools and the U.S. education 
system? 

□ Complete confidence 

□ A great deal of confidence 

□ Some confidence 

□ Very little confidence 

□ No confidence at all 

Source: Item adapted from the General Social Survey, retrieved from http://gss.norc.org/. 

8. What is the distance between your local public school and your current 
household? 

Source: Item adapted from the Spring Parent Interview instrument of the Early Childhood Longi- 
tudinal Study, Kindergarten Cohort of 2010-11, by the National Center for Education Statistics, 
retrieved from http://nces.ed.gov/ecls/. 


9. The amount of money spent on public schools varies greatly from community 
to community and state to state because some communities and states are richer 
than others and have more tax dollars to spend on education. It has been pro- 
posed that some revenues from richer states and communities be shifted to the 
school systems of poorer states and communities to provide those poorer states 
and communities with more resources for their schools. Would you favor or 
oppose such a policy? 

□ Favor 

□ Oppose 

Source: ABC News Education Poll, February 1990. Retrieved from Inter-university Consortium for 
Political and Social Research, retrieved from http://www.icpsr.umich.edu/icpsrweb/landing.jsp. 


10. Several different ideas have been proposed about how to teach children who don’t 
speak English when they enter our public schools. Please tell me which of the 
following programs you feel is the best approach to teach English learners. 

□ Two-way immersion programs 

□ Late-exit transitional programs 

□ Early-exit bilingual programs 

□ Something else 

Source: Item adapted from the General Social Survey, retrieved from http://gss.norc.org/. 


66 


Question-by-question follow-ups 


Ql 


Q2 


Q3 


Q4 


How much confidence do you have in your local schools and the U.S. education system? 

Probe: After the respondent answers the item , ask 

1. In your own words, what is this item asking? 

What does the word confidence mean to you? 

2. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 

What is the distance between your local public school and your current household? 

Have the respondent think out loud when answering this item and ask the following probes after the 
respondent answers the item: 

1. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 

Note the unit of measure that the respondent uses: for example, does the respondent answer in 
minutes away from the school or does the respondent respond in miles or blocks? 

The amount of money spent on public schools varies greatly from community to community and state 
to state because some communities and states are richer than others and have more tax dollars to 
spend on education. It has been proposed that some revenues from richer states and communities 
be shifted to the school systems of poorer states and communities to provide those poorer states 
and communities with more resources for their schools. 

Would you favor or oppose such a policy? 

Probe: After respondent answers the item , ask the following questions: 

1. In your own words, what is this item asking? 

2. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 

Several different ideas have been proposed about how to teach children who don’t speak English 
when they enter our public schools. Please tell me which of the listed programs you feel is the best 
approach to teach English learners. 

Probe: After respondent answers the item , ask the following questions: 

1. In your own words, can you tell me what each response category means? 

2. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 
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Exercise 2 


For cognitive interviewer : Protocol for conducting a cognitive interview for exercise 2 
( Note to interviewer: This script should he read to the respondent .) 

I would like to begin by thanking you for coming here today to help us. We’re scheduled 
to be here for 15 minutes, and I will make sure that we do not go beyond that time. We’ve 
asked you to come because your local school district is conducting a survey on residents’ 
attitudes about various education issues. Today, I’m going to ask you to fill out a survey and 
provide feedback so that we can help the school district make sure that all different types 
of residents will understand the actual survey. 

To help keep track of everything you say today, I’m going to audio record this session. I’ll 
also be taking notes, so you might notice me writing as we are talking. I want you to know 
that your responses are voluntary and confidential. 

In just a minute I’m going to hand you a short survey, and I’d like you to fill it out the same 
way you would if it came to you at home in the mail. Unlike the case of responding to a 
survey at home, though, I’ll ask you to stop periodically so that we can talk about what you 
were thinking as you answered the items. 

There also will be one item for which I will ask you to think “out loud.” For this item, it is 
important that you tell me what you are thinking and feeling while you are actually in the 
process of coming up with your answer. 

Now, because some people aren’t used to reading, thinking, and expressing their feelings 
aloud, I’d like to begin today with a short practice item. Please read out loud the item I’m 
about to give you and tell me your thoughts and feelings from the moment I handed you 
the item until you finish explaining your answer. 

(Note to interviewer: Give the practice item on the next page to the respondent for the think - 
aloud exercise .) 
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Question: How many times did you eat a meal at a restaurant in the past week? 
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Thank you. I’m going to hand you the sample survey now. Please read it and fill it out as if 
you were responding to the actual survey. As you go through the survey, if any item sticks 
out to you in any way (maybe you think an item is hard to answer, difficult to understand, 
located in a strange place, or you just don’t like it) — anything at all that you would like to 
say about an item — please tell me so that we can discuss it. 

(Note to interviewer: Give survey to respondent.) 
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For respondent: Survey for exercise 2 


The following survey will be mailed to a local school district. We are testing the items for 
comprehension and clarity. 

1. Do you think your child has an opportunity to obtain an education that corre- 
sponds to his or her abilities and talents? 

□ Yes 

□ No 

Source: Item adapted from the General Social Survey, retrieved from http://gss.norc.org/. 


2. How important do you think having well-educated parents is for getting ahead in 
life? 

□ Essential 

□ Very important 

□ Fairly important 

□ Not very important 

□ Not important at all 

Source: Item adapted from the General Social Survey, retrieved from http://gss.norc.org/. 


3. How important do you think having a good education yourself is for getting 
ahead in life? 

□ Essential 

□ Very important 

□ Fairly important 

□ Not very important 

□ Not important at all 

Source: Item adapted from the General Social Survey, retrieved from http://gss.norc.org/. 


4. How many times in the past year have you read about or watched a television 
program about topics related to education? 

times in the past year 

Source: New item created for exercise. 
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Question-by-question follow-ups 


Ql 


Q2 


Q3 


Q4 


Do you think your child has an opportunity to obtain an education that corresponds to his or her 
abilities and talents? 

Probe: After respondent answers the item , ask the following questions: 

1. In your own words, what is this item asking? 

2. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 

How important do you think having well-educated parents is for getting ahead in life? 

Probe: After respondent answers the item , ask: 

1. What does the phrase getting ahead in life mean to you? 

2. What about the term well-educated ? In your own words, what does this mean? 

How important do you think having a good education yourself is for getting ahead in life? 

Probe: After respondent answers the item , ask: 

1. What does the term good education mean to you? 

2. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 

How many times in the past year have you read about or watched a television program about topics 
related to education? 

Have the respondent think out loud when answering this item and ask the following probes after the 
respondent answers the item: 

1. What does the phrase in the past year mean to you? Were you thinking of since January or 
of the previous 12 months? 

2. Can you tell me that it was definitely this many times within the past year? 

3. Was this item easy or difficult for you to answer? 

If difficult , ask 

What made it difficult? 
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Module 5: Sampling 


Overview 


Duration: 30-45 minutes 
Slides: Module 5 slide deck 
Materials needed: 

• One copy of slides per participant. 

• Projector and computer. 

Agenda 


• Target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 


Background 


Defining the target population is an important first step in designing a survey project. 
A target population is the group of people or institutions (for example, schools) that the 
survey results are intended to represent. 

The goal of this module is for participants to understand how to select a sample of survey 
respondents that will represent the target population. 

Strategies for adapting module 5 to your context 


• No activity is provided for this module. However, as facilitator, you may want to 
develop a hands-on activity asking participants to select a sample for a project they 
are planning. They can discuss the following issues in small groups or the whole- 
group setting: 

o The definition of the target population. 

o The sampling frame and strategies for checking that the sampling frame is 
up-to-date and complete. 

o Whether a census of the whole target population or sample of it is most 
appropriate. 

o Whether to use random sampling and what subpopulations, if any, are appro- 
priate to use for stratified sampling. 

• If participants are not planning a survey, they can spend time in small groups 
considering a scenario in which district administrators would like to collect data 
about teachers’ experiences with professional development during the past three 
years. To collect the data, the administrators plan to email a survey link to prin- 
cipals in the district with a request that they forward the link to teachers in their 
schools so the teachers can take the survey. What issues might arise regarding the 
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target population, the sampling frame quality, and sampling strategies in this situ- 
ation? On the basis of the information covered in this session, what advice would 
participants offer the district? 

• Another possible activity would be to present the example in the notes to slide 18 
and then give participants time to discuss their answers to the following question: 
Which situation will yield the most representative data? Pairs or small groups can 
take a few minutes to work on their own; then everyone can report to the whole 
group. 
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Slides and notes for module 5 


Slide 1 




Module 5 
presentation 


Sampling 


Slide 2 



• Defining a target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 


Slide 3 



• Defining a target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 
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Slide 4 


Slide 5 


The target population is the group of people (or institutions) 
that the survey results will represent. For example, if the unit 
of analysis (as described in module 1) is the school, the target 
population is some set of schools. The target population will be 
determined by a research question, such as: 

• How safe do high school students feel at their school? 

• What barriers to school involvement do parents of English 
learner elementary school children experience? 

• What professional development activities are most prevalent 
among grade K-3 teachers ? 

Carefully define who is in your target population. 

• Teachers. Which grade levels? Which subject areas? What 
about specialists, special education teachers, full-time versus 
part-time teachers, long-term substitutes, or educational 
assistants? 

• Parents. Of all students? Can you include non-English-speaking 
parents? 

• Administrators. Principals and assistant principals? Other 
administrative positions? 

• Students. All grades? Prekindergarteners? Special education 
students? Students in alternative schools? 

The definition of the target population should be agreed on in the 
planning phase in order for it to align with the research questions 
(Fowler, 2009). 


Outline 


• Defining a target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 


Defining a target population 



Slide 6 


A sampling frame is a complete list of all eligible members of the 
target population. Ideally, the sampling frame has the following 
characteristics: 

• Includes every member of the target population. 

• Does not include members outside the target population (for 
example, retired teachers or students who have transferred). 

• Does not include duplicates (for example, two listings for 
parents with multiple students or for teachers who teach in two 
schools). 

• Is current. 


Start with 
a sampling 
frame. 


76 







Slide 7 


It is not necessary to survey every member of the target 
population (that is, everyone in the sampling frame). If fewer than 
1,000 members are in your target population (such as principals 
within a district), you may want to do a census and invite all 
members to take the survey. For larger target populations, highl- 
it IS not necessary to quality survey data can be obtained with a carefully selected 

Survey every member of sample of population members. 

the tarC|6t population. a random sample of sufficient size will elicit approximately the 

same overall results as a census of the target population. 



Random sampling is a powerful tool in survey research. A subset 
of target population members selected at random (where each 
member has an equal chance of being selected) can provide 

• Defining a target population. accurate data about the entire target population. Sample data 

• Sampling frame. allow statistical inferences about the population. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 


Outline 


Slide 9 


Simple random sample 


In a simple random sample, all members of a population have an 
equal chance of selection. A random sample can be selected in 
an Excel spreadsheet that includes the sampling frame as follows: 



• Create a column that calculates “=RAND()” for each member of 
the population (each row). 

• The RAND function returns random numbers between zero and 
one. 

• Sort rows by this random number. 

• If you want a sample of 500, take the top 500 rows. 


Slide 10 



Large random samples 
are more representative 
than smaller random 
samples. 


In general, the larger the sample, the better able you will be to 
detect statistical significance and draw inferences about the 
target population or census. Large random samples are preferable 
because they are more representative than smaller random 
samples. However, representativeness is more important than 
size for drawing inferences about the target population. That is, 
if you have the option of drawing a smaller random sample or 
a larger nonrandom sample, a smaller random sample is often 
preferable. 
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Slide 11 


Slide 12 


Slide 13 



Large random samples yield 
more precise estimates. 



How should you 
determine your sample 
size? 


Outline 


• Defining a target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 


A large random sample yields more precise estimates than a 
small random sample. For example, assume the proportion of 
all parents (your target population) who favor school uniforms is 
60 percent: 

• Ninety-five percent of the time, a random sample of 1,000 
parents will give you an estimate ranging between 57.0 percent 
and 63.0 percent. 

• Ninety-five percent of the time, a random sample of 500 
parents will give you an estimate ranging between 55.6 percent 
and 64.4 percent. 

There is a diminishing return on the precision benefits of a large 
sample size. Large sample sizes can be costly and may have 
implications for data quality; therefore, the benefits of increasing 
the sample size beyond what is needed for data analysis must be 
weighed against these costs. 

Ideal sample size depends on the following: 

• The importance of decisions that will be made on the basis of 
the survey results (a larger sample produces estimates with 
higher precision and confidence). 

• How many subgroups of interest there are (a large enough 
sample of each subgroup is needed for precise subgroup 
estimates). 

• The fact that not everyone sampled will participate (so 
sample size needs to be considered within the context of the 
anticipated response rate). 

• Budget. 

You may want to consider hiring a sampling statistician for large- 
scale, high-stakes surveys. Such a person can conduct power 
analyses to determine the appropriate sample size for your survey. 
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Stratified random sample 


Slide 14 


Slide 15 


Slide 16 
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Stratified for multiple 
characteristics 


Example 


Years of experience 


1-2 

3-10 

11+ 

Elementary 

8% 

20% 

10% 

Middle grades 

7% 

10% 

9% 

High school 

7% 

20% 

9% 


A stratified random sample forces the sample to match the 
target population on the distribution of important characteristics 
(such as region, race/ethnicity, or English learner status). It can 
increase the precision of your estimates compared with a simple 
random sample of the same size. 

A stratified random sample can be selected as follows: 

• Create separate sample frames for each subgroup. Randomly 
select the number of cases needed in each subgroup so that 
your sample will have the same percentage of members in each 
subgroup as the target population. 

• If, for example, your target population is 10,000 students 
(80 percent non-English learner students, 20 percent English 
learner students) and you want a sample size of 2,000: 

• Randomly select 1,600 from the non-English learner student 
frame. 

• Randomly select 400 from the English learner student frame. 
Stratification can be done for more than one characteristic. 

• Select a sample of teachers that matches the target population 
by school level and experience. 

• Determine cell percentages that match the target population 
(sum to 100 percent). 

• Create a separate frame for each cell, and sample with the 
frames so that the percentages within your final sample match 
the percentages in the target population. 

• Follow the method described in slide 9 to use Excel to randomly 
select the number of respondents needed in each frame to 
achieve the desired percentages in each cell. 


Outline 


• Defining a target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 


Slide 17 


Given the ease of sending survey invitations by email, it is 
tempting to invite every member of a large target population to 
take a survey. 



Should you 
conduct a 
census of a 
large target 
population? 


This approach can result in a large number of completed surveys 
but may not produce accurate survey data if the respondents are 
only a small fraction of the individuals invited to take the survey. 
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Slide 18 


Slide 19 



Example: If 40,000 high school 
students are in the target population, 
which situation will give you the most 
representative data? 


You randomly 


You attempt a 

sample 3,000 

OR 

census (40,000 

students to receive 

surveys are 

a survey. You end 


distributed), and 

up with 2,500 


you get 10,000 

completed surveys. 


completed surveys. 


Example: 

If 40,000 high school students are in the target population, which 
situation will give you the most representative data? 

• You randomly sample 3,000 students to receive a survey. You 
end up with 2,500 completed surveys. 

• You attempt a census (40,000 surveys are distributed), and you 
get 10,000 completed surveys. 

Receiving 2,500 completed surveys out of 3,000 distributed 
surveys is an 83 percent response rate. It might be hard to 
achieve that level of response, but it will be worth it in terms of 
data quality. 


Receiving 10,000 completed surveys out of 40,000 distributed 
surveys is a 25 percent response rate. Resources (such as the 
time required for staff to follow up with or track respondents 
or monetary incentives) are better spent on increasing the 
response rate of a smaller random sample than on administering, 
processing, and analyzing many surveys that may not be very 
representative. It is typically easier to achieve a high response 
rate with a smaller random sample than with the entire target 
population if that population is very large. 


It is possible that a sample of 2,500 might give you more 
representative data than a sample of 10,000. 


Those who easily agree to complete the survey may be very 
different in important ways from those who do not return a survey. 
For example, if the survey is gauging interest in afterschool 
programs, and students who have a strong interest in the 
programs are eager to respond to the survey, the results from 
the easily obtained 10,000 students would strongly overestimate 
interest. 



In general, the higher the proportion of respondents from the 
selected sample, the more representative the survey results 
are likely to be. Response rates are a key consideration for the 
representativeness of the sample. (More on this in module 7.) 


Representative samples 
are more important than 
large samples. 


Conducting a census of a very large target population is often 
not necessary to get accurate data about the target population. 
A large census should be attempted only if there are enough 
resources to ensure a high response rate. 


Slide 20 


Outline 


• Defining a target population. 

• Sampling frame. 

• Random sampling. 

• Stratified random sampling. 

• Attempting a large census. 

• Other considerations. 
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Slide 21 



Administering a survey at a convenient gathering of target 
population members (such as a football game or a parent-teacher 
association meeting), to individuals on a convenient contact list, 
or to individuals who have been responsive to surveys in the 
past — called convenience sampling — will likely lead to responses 
that are unrepresentative of the target population in important 
ways. For example, survey data collected from parents attending 
a parent-teacher association meeting should certainly not be 
considered representative of all parents at a school. 


Slide 22 
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Other sampling possibilities: 

• Intentionally oversampling rare subgroups to ensure an 
adequate sample size for separate estimates. 

• Randomly selecting schools and then randomly selecting 
students within those schools. 

These methods can be useful but require special technical 
operations, including creating weights for the data. If 
contemplating these methods, consider hiring a sampling 
statistician for large-scale, high-stakes surveys. 


Slide 23 


Additional resources 


Kalton, G. (1986). Introduction to survey sampling. Thousand Oaks, 
CA: SAGE. 

Kish, L. (1965). Survey sampling. New York, NY: John Wiley. 

Using Excel to make a random sample. Retrieved April 1, 2016, from 
http://www.voutube.com/watch?v=Q5aB3aX0z-E . 


For more information, please visit the following websites: 

• The American Association for Public Opinion Research: 
http://www.aapor.ora/ . 

• Public Opinion Quarterly: https://poa.oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.ora/content/current . 
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Module 6: Survey administration 


Overview 


Duration: 30-45 minutes 
Slides: Module 6 slide deck 
Materials needed: 

• One copy of slides per participant. 

• Writing tools. 

• Projector and computer. 

Agenda 

• Data collection modes. 

• Choose a mode. 

• Implementation. 

• Other considerations. 


Background 


This module describes the common modes for administering a survey and collecting data 
from respondents. A mode is how respondents are contacted, items are administered, and 
data are collected. 

The goal of this module is for participants to understand the strengths and weaknesses of 
each mode of data collection, the scenarios in which each mode is appropriate, and the 
logistics of survey administration. 

Strategies for adapting module 6 to your context 


• Many online survey tools are available. If participants are particularly interested in 
online tools, build in some time for each participant to explore one of the survey 
tools mentioned in the presentation (for example, Qualtrics, SurveyMonkey, 
Zoomerang, SurveyGizmo, WebSurveyor, Vovici, or Snap) and report to the group 
on its features, pricing, capabilities, and data storage issues as they relate to con- 
fidentiality concerns. Many of these tools have free trial services for a certain 
number of surveys and for students. 

• Another option for participant engagement is to develop scenarios related to your 
local context and ask participants to identify the mode of data collection they 
would use in each scenario. 
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Slides and notes for module 6 


Slide 1 
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Slide 2 


Outline 


• Data collection modes. 

• Choose a mode. 

• Implementation. 

• Other considerations. 



Slide 3 


Outline 


• Data collection modes. 

• Choose a mode. 

• Implementation. 

• Other considerations. 


Slide 4 



A mode is how sampled respondents are contacted (for example, 
in person, by telephone, by mail, or by email), how the survey is 
administered (that is, by an interviewer versus self-administered), 
and how data are collected from respondents (that is, on paper 
versus online; Klein, 2008). 

A survey can have multiple modes of data collection, such as a 
combination of computers and paper and pencil. Generally, modes 
are categorized as follows: 

• In person (personal interview). 

• Telephone (personal interview). 

• Paper and pencil (self-administered). 


• Online (self-administered). 
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Telephone and in-person 
surveys 


Slide 5 



In-person and telephone surveys tend to be more expensive than 
self-administered surveys. 

In-person interviews require paying a person to travel and 
interview respondents. 

Telephone surveys require some capacity to administer (such 
as a call center). Response rates for telephone surveys can be 
problematic because of the advent of Caller ID, which allows 
potential respondents to screen calls. Another important 
consideration for telephone surveys is whether phone numbers 
are available for the respondents from whom you wish to collect 
data. 


Slide 6 


Self-administered surveys 



Self-administered surveys (paper-and-pencil and online) are the 
most common modes. They are generally less expensive but must 
be set up so that the respondent can complete them without help. 

Respondents’ reading ability and language proficiency are key 
factors in designing a self-administered survey. Your pretesting 
protocol should address these concerns. Directions must be clear 
to eliminate respondent confusion, because no interviewer will be 
available to provide assistance. 

Response rates for online surveys can be lower than for surveys 
administered in person (Klein, 2008; Lavrakas, 2008). 
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Self-administered paper surveys 


What are the advantages of self-administered paper surveys? 

• They do not require knowledge of or access to computers and 
technology. 



• They can be created quickly. 

• They can be administered anywhere. 

What are the disadvantages of self-administered paper surveys? 

• They may require extensive time scanning data or individually 
inputting data into a database. 

• They are prone to respondent error if complex skips are 
included. 


• Paper forms can be lost. 


• Large-scale surveys are expensive. 

• Researchers may be unable to confirm that the survey was 
completed by appropriate respondent. 


(Ruddy, 2011) 
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Slide 8 


Slide 9 


Self-administered online 
surveys 




Advantages 

and 

disadvantages 



What are the advantages of self-administered online surveys? 

• No worries about collecting paper copies. 

• Responses are easily stored in a database. 

• Better options to navigate complex surveys with many skip 
patterns. 

• Faster responses. 

What are the disadvantages of self-administered online surveys? 

• Respondents need access to computers and the Internet, 
which may affect coverage or the number of people who can 
respond to the survey. Certain subpopulations (including, for 
example, geographic regions, elderly populations, and low- 
income households) may have variable access to and facility 
with computers and the Internet. 

• They take more time to develop than paper surveys. 

• Surveys that are administered via email only tend to have lower 
response rates. 

• Email addresses are not always available for all individuals 
within a target population. 

(Kendziora & Boccanfuso, 2011; Lavrakas, 2008; Ruddy, 2011) 

When choosing a mode for your study, consider the: 

• Types of respondents. 

• Survey complexity and item formatting. 

• Time available for administering the survey. 

• Resources, including money and staff expertise. 

• Location of survey administration. 
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Online versus paper? 


Slide 10 


Slide 11 


Slide 12 


Respondent 

type 



Online versus paper? 


Survey 
complexity 
and item 
formats 




Online versus paper? 


Timing of 

survey 

completion 



Considerations related to respondent type: 

• Students. 

° Paper. 

• Easy to administer in classrooms. 

• Students are familiar with the format. 

• Is scanning software, such as Scantron, available? 

° Online. 

• Can you administer in a school setting (for example, a 
school computer lab) so that computer access is not an 
issue? 

• If possible, try to administer the survey in a group setting 
(for example, in homeroom) to boost response. 

• Online may be a good option for sensitive topics in order to 
preserve respondents’ privacy. 

• Staff. 

° Paper — can do at home or at school. 

° Online — can do at home or at school. 

• If staff share computers, this will be a consideration for 
certain kinds of online surveys that require different users 
to access the survey from different computers. 

• Families/guardians. 

° Paper — how will the survey be returned? May need stamped, 
preaddressed envelopes for mailing back or a location at the 
school where families can return them. 

° Online — computer access or familiarity may be an issue. May 
need to send a link to the survey by mail and email. 
Considerations related to survey complexity and item format: 

• How complex is the survey? 

• Does it contain many skip patterns? 

• What item formats do you want to include? 

• Online surveys allow for a very complex survey structure that 
will be seamless to the respondent. 

• Paper surveys that require the respondent to skip items can be 
difficult for some respondents to complete. If directions are not 
followed as expected, inconsistencies will need to be corrected. 

• If your survey includes many open-ended items, an online 
survey will be preferred. Typed responses on an online survey 
will be captured, automatically saving data-entry costs. Having 
to decipher illegible responses is avoided in an online survey. 

Factors that may impact the timing of survey completion: 

• When will the survey be conducted? The more complex the 
survey, the longer it will take to program and test for online 
surveys, which can possibly delay when administration begins. 

• Paper surveys that require the respondent to skip items may 
be difficult for the respondent to complete and may require 
additional editing time after you receive the final data file. 

• Paper surveys generally are quick to launch but take longer to 
process, whereas online surveys can involve more time up front 
but are quicker on the back end. 
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Slide 13 


Online versus paper? 


Resources 



Considerations related to resources: 

• Are staff members available to program an online survey? 
Making online surveys compatible with tablets and 
smartphones is another programming consideration. 

• Will you need scanning capability (for example, Scantron) for 
paper surveys? 

• Larger-scale projects will require more resources regardless of 
the mode. 
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Online versus paper? 


Location 



Considerations related to location: 

• Paper surveys are easier to administer in a group setting. If 
you already have the target population in a group setting (for 
example, participants in a professional development session), 
you can administer a paper survey while everyone is in the 
room, which is likely to achieve a higher response rate than 
having participants log in to an online survey after the training. 

• School-based surveys can take advantage of school computer 
resources (such as a computer lab) to administer an online 
survey in a group setting. 
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Outline 


• Data collection modes. 

• Choose a mode. 

• Implementation. 

• Other considerations. 


Slide 16 


Planning for an online survey 



Start planning early. 

Identify a survey coordinator who will: 

• Work with the information technology staff if necessary to 
manage survey invitations. 

• Coordinate the survey administration and act as the point of 
contact. 

• Monitor response rates. 

• Follow up with nonrespondents. 

(Kendziora & Boccanfuso, 2011; Ruddy, 2011) 
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Planning for a paper and 
online survey 



Outline 


• Data collection modes. 

• Choose a mode. 

• Implementation. 

• Other considerations. 


Identify a survey program if administering online. Some districts 
have data systems (for example, Naviance, eSchool, and Aesop). 
There also are several commercially available online software 
programs. Commercial online survey programs include: 

• Qualtrics. 

• Survey Mon key. 

• SurveyGizmo. 

• WebSurveyor. 

• Vovici. 

• Snap. 

Allow time for development and multiple rounds of testing. 

Select a time to implement the survey: 

• Depending on the purpose of the survey, choosing the end 
of the school year or semester in education settings may be 
preferable. 

• Avoid conflicts with other surveys and school and district 
activities (for example, the assessment calendar). 

• For an online survey — ensure that computers or computer labs 
will be available. 

• For a paper survey — allow additional time for printing and 
entering data into a database. 

Identify ways to “get the word out” about the survey, such as 
newsletters or meetings. 

Identify respondents who might need special accommodations, 
such as those with low English proficiency, low literacy, or 
special needs. Schools usually facilitate accommodations during 
standardized testing for these students, and you might be able 
to use the same accommodations to administer surveys (Ruddy, 
2011 ). 

Other considerations relate to informed consent and data 
confidentiality. 
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Many schools and districts have procedures for obtaining 
permission from parents for their children to participate in 
data collection activities and guidelines on what information 
participants should be given before participating in a study. 
Generally, participants should be told how long the study is 
expected to last, that participation is voluntary, and how their 
personal information and responses will be protected and used. 


88 







Slide 21 


Slide 22 



Some schools or districts require parent permission for student 
participation in surveys. Survey information is sent to parents, 
and a signed form is required for the student to participate. For 
example, the letter requesting parent consent could include such 
language as “If you agree to let your son or daughter participate, 
please sign and return the consent form enclosed with this letter 
by [month day, year].” 

In these districts, parents who do not respond to a request 
for permission are assumed to be declining to let their child 
participate in a survey. 

Other districts require only parent notification for students’ 
participation in surveys. In this case, parents are asked whether 
they would like their child to opt out of the survey. If a parent does 
not respond, it is assumed that the child can participate if the 
child assents. 


Survey information and permission and notification forms should 
be provided in the parents’ home language and easily understood. 
If permission forms need to be signed, researchers should retain 
copies for their records. 

It is important to find out what requirements are applicable at all 
schools in advance to allow time for this process. 


Key components of 
informed consent forms 


• Study purpose. 

• How data will be used. 

• Voluntary participation. 

• Burden. 

• Risks. 

• Benefits. 

• Confidentiality. 

• Contact information. 

• Documentation of consent. 


Informed consent forms should include the following components: 

• The purpose and benefits of the study (for example, “The survey 

results will be used to gain understanding about ,” or “The 

survey results will be used to inform decisions about ”). 

• How the data will be stored and used. 

• That participation in the study is voluntary. 

• How long the survey will take. 

• Risks (for example, “No anticipated or known risks are expected 
in participating in this study.”). 


• Privacy (for example, “All information we collect for the study 
will be used for research purposes only and will be kept 
confidential.”). 


• A person to contact for more information. 


• Documentation of consent. 


Such consent forms ensure that individuals are voluntarily 
participating in research by making a fully informed decision. 
Parents are asked for permission to include their children in 
research studies. Children are asked to give their assent to 
participate. 

The language used in an informed consent form should be clear, 
fair, and at an appropriate level for the individual’s ability to 
understand the information being presented. If needed, consent 
forms should be translated into languages appropriate for the 
intended research audience. 


Remember, informed consent does not consist of simply 
getting signed consent forms; it is a communication process by 
which project staff members reach agreement with people to 
participate in their projects. Its ethical intent is to communicate 
clearly and respectfully; foster trust, comprehension, and good 
decisionmaking; and ensure that participation is informed and 
voluntary. 

Entities such as institutional review boards (for example, the 
Institutional Review Board of American Institutes for Research) 
and the Office for Human Research Protections aid in the 
determination of whether project staff are behaving ethically. 
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Ensure 
data security 
and 

confidentiality. 


Additional resources 


For more information, please visit the following websites: 

• The American Association for Public Opinion Research: 
httD://www.aaDor.ora/ . 

• Public Opinion Quarterly: https://Doa.oxfordiournals.ora/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.orq/content/current . 


• Data security issues: 

° Paper — instead of using respondent names or other 
information that could be used to personally identify a 
respondent, use barcodes or unique identification codes to 
track who has responded to the survey. 

° Online — provide users with logins. Consider using passwords 
so that respondents can return to the survey if they are 
interrupted. 

• Confidentiality: 

° A confidential survey collects but does not report any 
identifying information. 

° If identifying information is collected, describe how it will be 
stored separately from survey responses (for example, linked 
only by an identification number). 

° Data should be reported in aggregate so that specific people 
cannot be identified. 

° Confidential and anonymous are not the same thing. 

An anonymous survey does not collect any identifying 
information. If survey administrators or data collectors can 
identify the respondent in any way, it is not an anonymous 
survey. 
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Module 7: Response rates 


Overview 


Duration: 45-60 minutes 
Slides: Module 7 slide deck 
Handouts: 

• Handout 7d: Guidelines for using incentives in surveys- 

• Handout 7.2: Examples of emails to respondents. 

• Handout 7.3: Sample report on potential nonresponse bias. 

No activity is provided for this module, but the three handouts offer guidance and can be 
mentioned during the presentation as helpful resources. If time allows and the participants 
have a particular interest in incentives (handout 7.1), communicating with respondents 
(handout 7.2), or investigating and reporting potential nonresponse bias (handout 7.3), 
these materials can be incorporated into the session. Participants can be given time to 
read a handout and ask questions or make observations to generate discussion. Encourage 
participants to draw from their own experiences and contexts to make the discussion more 
relevant. 

Materials needed: 

• One copy of slides and handouts 7.1, 7.2, and 7.3 per participant. 

• Writing tools. 

• Projector and computer. 

Agenda 


• What are response rates? 

• Strategies for maximizing response rates. 

• Nonresponse bias. 

• Summary of best practices. 


Background 


The response rate is the percentage of eligible individuals in the sample who complete a 
survey. A survey s response rate is a key indicator of quality because low response rates can 
severely limit the representativeness of the survey data and therefore limit the usefulness 
of the survey results. The goal of this module is for participants to learn how to calculate 
response rates, maximize response rates, and investigate bias caused by nonresponse. 

Strategies for adapting module 7 to your context 


• Use surveys that someone in the group has administered to calculate response 
rates. Have a group discussion about what response rates the group would find 
acceptable and strategies they might have used to improve response rates. 
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Slides and notes for module 7 
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Module 7 
presentation 

Response 

rates 


April 2016 


Slide 2 


Outline 


• What are response rates? 

• Strategies for maximizing 
response rates. 

• Nonresponse bias. 

• Summary of best practices. 
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Outline 


• What are response rates? 

• Strategies for maximizing 
response rates. 

• Nonresponse bias. 

• Summary of best practices. 
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The response rate is the percentage of people who complete 
and return a survey out of the total number of eligible sample 
members. It is a measure of the level of success achieved in 
collecting survey data (American Association for Public Opinion 
Research, 2009a). 
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Calculating response rate 


Response rate = 


# of completed surveys 
# of eligible units sampled 


The response rate is calculated by dividing the number of 
completed surveys by the total number of eligible sample 
members (American Association for Public Opinion Research, 
2009a). For example, if a survey is sent to a teacher who you later 
learn has retired and your target population is “current teachers,” 
that teacher is not eligible and not included in the denominator. 
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Calculating response rate 


For example: If you send out 
200 questionnaires and you get 
125 questionnaires back, your response rate 
is 62.5%. 


Response rate = 125 x iqo = 62.5% 
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The response rate is one way to gauge whether survey results 
represent the target population: 

• A high response rate maximizes the chance that the results are 
representative of the target population. 

• A low response rate increases the chance of biased results, 
which cannot be generalized to the target population. 



No standard target response rate exists. Higher response rates 
typically (but not always) mean that data better reflect the target 
population. 



Is there a 

standard 

target 

response 

rate? 


Various federal agencies have standards for data collection 
response rates. For example, the National Center for Education 
Statistics has a response rate target of 85 percent. In some 
contexts, a response rate in the 70 percent range is considered 
adequate. Achieving a high response rate should be a priority for 
any survey project (Groves, 2006). 
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Outline 


• What are response rates? 

• Strategies for maximizing 
response rates. 

• Nonresponse bias. 

• Summary of best practices. 
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What 

influences 

response 

rates? 


Response rates are influenced by: 

• Salience of the topic — interest of respondents in the topic of 
the survey. 

• Personalized requests and communications. 

• Multiple follow-up contacts. 

• Respondents’ respect for and trust in the survey sponsor. 

• Conciseness and easiness of the survey. 

• Data collection mode. 

(Dillman, 2000) 



How do you maximize response rates? 

• Generate positive publicity for your survey. 

• Send an endorsement letter from a respected leader. 



How do you 
maximize 
response 
rates? 


• Appeal to people’s tendency to help — ask them to help by 
giving their input. 

• Make the survey topic salient: 

° Ensure that respondents see the value of the survey and 
their responses. 

° Point out respondents’ personal connection to the topic. 

° Tailor and personalize communications about the survey and 
its purpose. 


• Make the survey attractive and easy to complete and return. 


• Provide clear instructions about how to complete and submit 
the survey. 

• Show positive regard; say thank you. 

• Ensure the confidentiality of the information provided. 

• Provide an incentive (a token of appreciation). 


• For a mailed survey, provide return postage. 


• Follow up several times by mail, email, telephone, or in person. 


• Use a combination of survey modes (for example, telephone 
plus mail, or online plus mail). 


(Dillman, 2000) 
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Personalize communications 
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Incentives 



To motivate people to respond, appeal to their tendency to help. 

• Use the person’s name and, if appropriate or possible, 
personalize salutations. 

° Impersonal: “Dear principal.” 

° Personal: “Dear Principal Lopez.” 

• Personalize the importance of the survey to the respondent. 

° Impersonal: “This survey will help us better understand the 
importance of instructional coaching for elementary school 
reading specialists.” 

° Personal: “This survey will provide information that can help 
better support you as a reading specialist and enable you to 
better support your students.” 

• Use recognizable logos or graphics. 

• Tailor the materials to a school or district, if appropriate or 
possible, using local letterhead or colors. 

(Scheuren, n.d.) 

Research shows that incentives increase response rates (see 
handout 7.1; Dillman, 2000). In general, a prepaid incentive works 
better than a promised reward for completing the survey. 

Types of incentives: 

• Cash (most effective). 

• Nonmonetary gift (for example, a coffee mug or a pen). 

• Lotteries — for example, respondents are promised a chance of 
winning a prize (least effective). 
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Handout 7.1 
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Contact participants 



Consider making the following contacts: 

• Brief prenotification (in online surveys this is particularly useful 
because bounceback notifications of undeliverable emails allow 
time to update the contact list before the survey invitations are 
sent). 

• Survey mailed with a cover letter or survey link sent in an 
invitation email. 

• Thank you follow-up. 

• Reminder with replacement survey or survey link. 

• Final contact: reminder with replacement survey or survey link. 


• Thank you note. 


(See appendix A; Dillman, 2000; Scheuren, n.d.) 
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Handout 7.2 
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Follow-up 



Slide 18 


Outline 


• What are response rates? 

• Strategies for maximizing 
response rates. 

• Nonresponse bias. 

• Summary of best practices. 


Nonrespondents are individuals who were randomly chosen for the 
sample and given the survey but did not return it. Failure to follow 
up with nonrespondents can ruin an otherwise well-designed 
survey (Dillman, 2000; Scheuren, n.d.). 

About a week or two after sending the survey, send a gentle 
reminder to anyone who has not responded. Allow an adequate 
amount of time between the survey being sent out and the time it 
would take someone to respond. The follow-up timeline differs by 
mode. For phone or online surveys, follow up within a week. For 
mailed surveys, follow up within two weeks. The reminder can be 
by email, postcard, or phone call. 
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Who are your 
nonrespondents? 


& & & Z 
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A potential for nonresponse bias exists if responders and 
nonresponders differ on characteristics that are related to the 
topic of interest. Responders and nonresponders may differ in 
many ways, but as long as the differences are not related to the 
topic of interest, they should not result in nonresponse bias. 

Flowever, if these differences are related to the topic of interest, 
nonresponse bias could arise (Groves, 2006). 

Nonresponse bias harms the validity of the survey. It weakens the 
ability to draw accurate conclusions about the target population. 

• If the characteristics of survey respondents and 
nonrespondents that are different are also related to the topic 
of interest, there may be bias, and the data should not be 
generalized even if the response rate is relatively high. 


• If the characteristics of the survey respondents are the same 
as the nonrespondents, it may be okay to generalize the results 
to the population even if the response rate is low. 


• Even if all measured characteristics are the same for 
respondents and nonrespondents, there may be nonresponse 
bias related to unmeasured differences. 
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Measure 

nonresponse 

bias. 



Nonresponse example 


Bias analysis 


Percent 

Teacher Total eligible Completed survey Difference relative 

characteristic teachers (%) respondents (%) (%) difference 


Grade level 

Elementary 45 65 +20 20/45 = 44% 

Secondary 55 35 -20 20/55 = 36% 


How can nonresponse bias be measured? Assess the degree 
to which your survey respondents differ from the people in the 
sampling frame on key characteristics. Ideally, the sampling frame 
includes some key characteristics on which you can compare 
respondents with nonrespondents. For example, a sampling frame 
of teachers may include each teacher’s grade level, tenure, and 
gender (Groves, 2006). 

Nonresponse bias cannot be truly measured because the true 
value of what is being estimated is rarely known. However, the 
possibility of nonresponse bias can be measured. 

Example: 

• Two hundred elementary school and secondary school teachers 
need to be contacted for a survey. 

• The secondary teachers have annual assessments at the same 
time and are less likely to answer the survey. 

• Because of a response rate of 35 percent of secondary 
teachers, the survey data may not be representative 

of all teachers because secondary teachers will be 
underrepresented. 

• Compare the percentage of respondents who teach elementary 
and secondary school grades with the percentage of 
elementary and secondary school teachers in the sampling 
frame. 

• If the percentages are notably different, and you think that 
elementary and secondary school teachers differ on the topic 
of interest (for example, methods for addressing discipline 
problems), you likely have nonresponse bias. The degree of 
bias reflects the degree to which the topic of interest relates to 
grade level. 

Bias analysis: 

• Look at the difference between the percentage distributions 
of the eligible teachers and the responding teachers, by 
characteristic. 

• Difference can be expressed as percent relative difference, 
which is the difference between the eligible percentage and 
the respondent percentage, divided by the eligible percentage, 
and multiplied by 100. In this case, the relative differences 
are 44 percent for elementary teachers and 36 percent 

for secondary teachers, which suggests that bias may be 
occurring. 

• If you think that elementary and secondary school teachers 
differ on the topic of interest (for example, importance of 
teacher collaboration), a difference in their response rates 
may affect the validity of the results. The presence of 
nonresponse bias depends on whether a link exists between 
the characteristics of the respondents and nonrespondents and 
the topic of interest. 


97 






What if large differences between characteristics of respondents 
and nonrespondents are found? 

• Describe the results in terms of who did respond. 

• Do not imply that the results apply to anyone other than those 
who responded. 

• Specify who responded (that is, the group the data represent). 

• Report results by subgroups for which there were large 
differences in response rates (for example, by elementary and 
secondary school teachers). 

• Consult a statistician about other options for addressing 
nonresponse bias. 

(Scheuren, n.d.) 
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Address 
potential bias. 
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If the response rates are low (for example, lower than 70 percent), 
use a summary statement in reporting results, such as: “The 
response rate was only 42 percent, so care should be taken 
when reviewing these results. The results should be applied 
to individuals who responded, not to the entire population of 
teachers” (Scheuren, n.d.). If a nonresponse bias investigation 
suggests that the responders and nonresponders were similar 
on certain characteristics, this could be noted as evidence about 
the potential representativeness of data from a study with a low 
response rate. 
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Outline 


• What are response rates? 

• Strategies for maximizing 
response rates. 

• Nonresponse bias. 

• Summary of best practices. 
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What are the best practices regarding response rates? Follow the 

strategies presented here to maximize the response rate: 

• Monitor response rate overall and within groups of interest 
during data collection, and target additional follow-up contact if 
warranted. 

• Calculate and report the response rate. 

• Examine the data for nonresponse bias. 

• Discuss the survey results in the context of the response rate 
and any possible nonresponse bias identified. 

(American Association for Public Opinion Research, 2009b) 
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Additional resources 


For more information, please visit the following websites: 

• The American Association for Public Opinion Research: 
http://www.aapor.org /. 

• Public Opinion Quarterly: https://poa. oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.org/content/current . 
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Handout 7.1: Guidelines for using incentives in surveys 


• An incentive is one tool that can maximize response rates- An incentive’s effec- 
tiveness should be determined with respect to its cost relative to the reduction of 
nonresponse bias and increases in response quality- If surveying within an organi- 
zation, check policies related to staff accepting incentives. 

• Rather than using differential incentives, give a small incentive to everyone at 
initial contact and offer an additional incentive only to those who refuse the first 
request. 

• Because most respondents do not fall into many survey samples, the conditioning 
effect of giving an incentive will be minimal- But interviewees who take numerous 
surveys that employ incentives may come to expect them. 

• Prepaid incentives are more effective than incentives for which receipt is contin- 
gent on completing a survey- Prepaid incentives used for refusal conversion are 
much more effective than promised incentives. 

• Lotteries are generally not as effective at increasing response rates as certain 
receipt of an incentive- Lotteries are often used for online surveys of students, but 
the effect of lotteries is usually minimal. Students often prefer smaller prizes with a 
greater chance of winning to larger prizes with a lesser chance of winning. 

• Cash is the most effective incentive for increasing response rates in mail surveys, 
and monetary incentives are more effective than nonmonetary incentives, even 
controlling for value. Small, nonmonetary incentives are not worth the cost and 
effort given their likely minimal effects on response rates. Donations to charities 
have little, if any, positive effect and can have a negative effect. 

• Incentives represent one method, among a number of methods, used to gain initial 
cooperation and reduce attrition in panel surveys- A monetary incentive is the 
most effective, and its size matters more in panel surveys (surveys that are repeated 
over time with the same sample) than it does in cross-sectional studies (surveys 
that are conducted once). An incentive in the first wave usually has carryover 
effects and often does not have to be repeated. 

• A larger monetary incentive usually has a greater effect than a smaller incentive, 
but the rate of increase in the response rate declines as the incentive increas- 
es. The most commonly used initial incentive is $1 or $2. A larger amount (for 
example, $5) may be needed for refusal conversion, but the greater ease of refusal 
conversion using an incentive must be weighed against the cost of the incentive. 

• The level of respondent burden, the likely response rate without an incentive, 
and the desired response rate play roles in determining the appropriate size of the 
incentive. With respect to an incentive’s overall contribution to the cost of field- 
ing a survey, the cost of a single completion when not using an incentive will be 
particularly important. Even a small incentive can have a large effect on the cost 
of fielding the survey. 

Source: Tucker & Medway (2013). 
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Handout 7.2: Examples of emails to respondents 


One of the best ways to ensure a high response rate to a survey is to make frequent contact 
with your potential respondents. 

Example 1 

The following emails were sent to encourage Iowa principals to participate in a survey. 

First contact : Prenotice letter 

Subject: Important Principal Survey Coming Soon 

[Insert date] 

Dear [insert position or name]: 

In a few days, you will receive an email request to complete a brief survey for an important 
research initiative conducted by the School Administrators of Iowa and the Administrator 
Professional Learning Providers of Iowa. 

The survey will ask you to comment on your recent professional learning experiences and 
your experiences implementing teacher learning teams in your school. Your responses will 
provide critical information about the activities and views of principals, which will be used 
to inform a new statewide principal-training program aimed at building principals’ collec- 
tive capacity to improve student achievement in schools. 

Thank you for your time and participation. Your feedback is very important to us. It is only 
with the generous help of Iowa principals that our research can be successful. 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Second contact: Cover letter 


Subject: Professional Learning Survey Invitation 
[Insert date] 

Dear [insert position or name]: 

I am writing to request your help in a study of professional learning experiences of Iowa 
principals and the use of teacher learning teams in their schools. The School Administra- 
tors of Iowa (SAI) and the Administrator Professional Learning Providers of Iowa (APLPI) 
are conducting the study. Data from this survey will help SAI and APLPI better under- 
stand principals’ experiences interacting with and learning from executive coaches and 
implementing teacher learning teams in their schools. Your feedback is very important 
to us; it will help us develop plans for a prospective statewide training program that offers 
individualized leadership coaching to principals. Coaching will focus on helping principals 
build the collective capacity of their schools through the implementation of teacher learn- 
ing teams. 

Your responses are completely confidential. Only the researchers from an external research 
organization will have access to the responses, and results will be shared only as summa- 
ries; individual answers will not be identifiable. The survey is brief and should take only 
about 10 minutes to complete. To begin, please use the following link: 

GO TO SURVEY -► [insert hyperlink]. 

We know that principals frequently are asked to complete surveys and share their views. 
As a fellow educator, please consider the benefits this study has in shaping the principal- 
ship and schools in Iowa. If you have any questions about the content of the survey, please 
contact me at [insert email address]. If you experience any technical difficulties with the 
survey, please contact [insert name] at [insert email address]. 

Thank you very much for helping with this important work. 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Third contact (follow-up 1): Thank you and reminder 


Subject: Professional Learning Survey Reminder 
[Insert date] 

Dear [insert position or name]: 

Last week, you received an email asking you to complete a survey about your professional 
learning experiences and the use of teacher learning teams in Iowa schools. If you have 
already completed the survey, please accept my sincere thanks. If not, please do so today. 
The survey is brief and will take only about 10 minutes to complete. To begin, please use 
the following link: 

GO TO SURVEY -► [insert hyperlink]. 

We truly appreciate your time and participation. It is only by you sharing your ideas and 
experiences that we can develop high-quality and relevant professional learning experi- 
ences for Iowa principals. If you have any questions about the content of the survey, please 
contact me at [insert email address]. If you experience any technical difficulties with the 
survey, please contact [insert name] at [insert email address]. 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Fourth contact (follow-up 2): Final contact 


Subject: Final Chance to Participate in Professional Learning Survey 
[Insert date] 

Dear Iowa Principal: 

During the last couple of weeks, we have sent you email requests regarding an Iowa prin- 
cipal survey. If you have already completed the survey, please accept my sincere thanks. 
This very brief 10-minute survey is an important element of a research initiative we are 
conducting to better understand principals’ attitudes and thoughts about professional sup- 
ports and teacher learning teams. In addition, the survey will help us identify the barriers 
and challenges in building collective capacity to improve student achievement in schools. 

We are sending the survey link one last time to ensure that all Iowa principals who have 
not responded have an opportunity to do so. In addition, those principals who have not 
responded may have different experiences, thoughts, and attitudes than those who have 
responded. Hearing from all Iowa principals helps ensure that the survey results are as 
accurate as possible. 

Your responses are completely confidential and will be released only as summaries in which 
no individual’s answers can be identified. To begin, please use the following link: 

GO TO SURVEY -► [insert hyperlink]. 

We truly appreciate your time and participation. It is only with the generous help of Iowa 
principals that our research can be successful. If you have any questions about the content 
of the survey, please contact me at [insert email address]. If you experience any technical 
difficulties with the survey, please contact [insert name] at [insert email address]. 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Example 2 


These emails were sent to persuade district hiring officers in Minnesota to participate in a 
recurring survey. 

First contact: Prenotice letter 

Subject: Important Survey Coming Soon 

[Insert date] 

Dear [insert name]: 

It is time again for [insert survey name]. [Insert a few sentences about the survey and any 
relevant information that could be relayed to the respondent.] 

We are mindful of the data burden schools face and have made the survey much shorter 
than in previous years. It should take about 30 minutes to complete. 

The due date for this survey is [insert due date]. Please watch for the survey to arrive in 
your email inbox next week. 

Anytime you have questions, please email [insert contact name] at [insert email address]. 
Thank you very much for helping with this important work! 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Second contact: Cover letter 


Subject: Survey Invitation 
[Insert date] 

Dear [insert name]: 

[Insert a few sentences about the survey s purpose.] 

GO TO SURVEY — ► [insert survey link] 

Login: [insert login] 

Password: [insert password] 

Please complete the survey by [insert due date]. We would like a 100 percent response rate 
so that we get an accurate picture of [insert topic]. We are mindful of the data burden 
schools face and have kept the survey as short as possible. The survey should take no more 
than 30 minutes to complete. [Replace previous two sentences with relevant information.] 

You can access the Web survey from multiple computers by using the login and password 
included in this email. All responses are saved as you complete each page in case you have 
to close the survey and come back to it later. 

Results from the survey will be presented in aggregate in the final report. 

If you have questions, contact [insert contact name] at [insert email address]. 

Thank you very much for helping with this important work! 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Third contact (follow-up 1): reminder 


Subject: Survey Reminder 
[Insert date] 

Dear [insert name]: 

We are sending you this friendly reminder to make sure someone from your school or dis- 
trict responds to the [insert name of survey]. The link to the electronic survey and the 
login and password information are as follows: 

GO TO SURVEY -► [insert survey link] 

Login: [insert login] 

Password: [insert password] 

The following are answers to questions you might have: 

• What is the due date? Please complete the survey by [insert date in bold]. The 
survey should take less than [insert time in bold] minutes to complete, 

• How will the survey information be used? [Insert a brief description about how the 
information will be used,] 

• Why another survey? [If needed, use this space to explain the importance of this 
survey,] 

• Will my responses be singled out? [Use this space to address confidentiality issues.] 

• Who do I contact with questions? Contact [insert contact name] at [insert email 
address]. 

Thank you very much for helping with this important work! 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Fourth contact (follow-up 2): Final contact 


Subject: Final Chance to Participate in Important Survey 
[Insert date] 

Dear [insert name]: 

We noticed that you have not yet had a chance to complete the [insert survey name]. We 
recognize how busy you must be at this time of year and have extended the deadline for 
this survey to [insert date in bold]. It is important that we hear from everyone to ensure 
that we have the most accurate information. The survey is expected to take about 30 
minutes. 

GO TO SURVEY — ► [insert survey link] 

Login: [insert login] 

Password: [insert password] 

The following table represents the response rates we have, by region. Please respond soon 
so that your region is fully represented in the results of the survey. 


1 Region 

Response Rate I 

West 

72% 

East 

70% 

South 

87% 

North 

82% 

Central 

88% 


Please contact [insert contact name] at [insert email address] if you have any questions. 
We appreciate your prompt cooperation. 

Best regards, 

[Insert signature] 

[Insert name, affiliation, and contact information] 
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Handout 7.3: Sample report on potential nonresponse bias 


This excerpt from an unpublished report describes the potential for nonresponse bias. It 
can be used as a model for reporting a nonresponse bias analysis. 

The sampling frame consisted of the 1,218 principals. The frame included email 
addresses for principals from all elementary, middle, junior, and senior high 
schools; special education, alternative, and charter schools; plus early childhood 
centers. 

We sent email invitations to all these principals to take the online survey. Surveys 
were returned by 707 principals, for a 58 percent response rate. 

To compare the characteristics of the schools of our survey respondents with char- 
acteristics of the target population of all schools in the state, we looked at an exist- 
ing data source about all state schools. We used 2010/11 data from the National 
Center for Education Statistics (NCES) Common Core of Data, which had data 
from 1,164 public schools to compare the characteristics of responding schools to 
the known characteristics of all schools, including those that were nonrespon- 
dents (our sampling frame and the population of schools from the 2010/11 NCES 
data file differed by 54 schools). The table below shows the percentage distribution 
of some defining school characteristics of responding and nonresponding schools. 

The sample of schools for which we have survey data is quite similar to all schools 
in the sampling frame in terms of school size, school type, and area education 
agency. However, principals in city schools and, relatedly, Title I schools with rela- 
tively high minority enrollment are somewhat underrepresented in this survey. For 
example, 18 percent of schools in the sampling frame are located in a city, but only 
14 percent of the survey respondents are principals of city schools. 

The findings presented in this report should be interpreted with the understanding 
that not all types of schools are represented in the same proportions as they appear 
in the full population. See the table below for more details about the respondents 
and nonrespondents to this survey. 
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Number and percentage of schools participating and not participating in the survey, by school 
characteristics 


School 

All schools in the state, 
Common Core of Data 
( N = 1,164) 

Responding 
schools 
(n = 707) 


Nonresponding 
schools 
(n = 457) 

Difference between 
respondents and 
all principals 

characteristic 

Number Percent 

Number Percent 

Number Percent 

(percentage points) 

Locale 

City 

210 

18 

101 

14 

109 

24 

-4 

Suburb 

78 

7 

51 

7 

27 

6 

0 

Town 

278 

24 

182 

26 

96 

21 

2 

Rural 

586 

51 

369 

52 

217 

48 

2 

School size (number of students) 

0-500 

916 

80 

563 

80 

353 

79 

1 

501-1,000 

189 

16 

113 

16 

76 

17 

0 

1,001-1,500 

30 

3 

15 

2 

15 

3 

0 

1,500+ 

16 

1 

12 

2 

4 

1 

0 

Area education agency 

1 

75 

6 

42 

6 

33 

7 

1 

5 

93 

8 

46 

7 

47 

10 

2 

7 

169 

14 

106 

15 

63 

14 

0 

9 

101 

9 

66 

9 

35 

8 

-1 

10 

150 

13 

90 

13 

60 

13 

0 

11 

254 

22 

151 

21 

103 

23 

1 

12 

102 

9 

69 

10 

33 

7 

-2 

13 

125 

11 

83 

12 

42 

9 

-2 

15 

94 

8 

54 

8 

40 

9 

1 

School level 

Elementary 

596 

52 

355 

50 

241 

54 

-1 

Middle 

235 

20 

144 

20 

91 

20 

0 

High 

282 

25 

182 

26 

100 

22 

1 

Other 

39 

3 

22 

3 

17 

4 

0 

Title 1 school? 

No 

359 

31 

239 

34 

120 

27 

3 

Yes 

415 

36 

232 

33 

183 

41 

-3 

Not applicable 

378 

33 

232 

33 

146 

33 

0 

Percentage of students who are a racial/ethnic minority 

0-5 

348 

30 

209 

30 

139 

31 

0 

6-25 

596 

52 

394 

56 

202 

45 

4 

26+ 

205 

18 

98 

14 

107 

24 

-4 



no 









Module 8: Focus groups 


Overview 


Duration: 45 minutes, including 15 minutes for activity 8d 

Slides: Module 8 slide deck 

Handouts: 

• Handout 8d: Planning for a focus group, 

• Handout 8,2: Sample moderator guide for a structured focus group conducted to 
pretest survey items. 

Materials needed: 

• One copy of slides and handouts per participant. 

• Writing tools. 

• Projector and computer. 

Agenda 


• Overview. 

• Planning. 

• Recruiting. 

• Moderating. 

• Analyzing and reporting. 

• Activity 8.1. 


Background 


A focus group is a qualitative research method that can be used to formulate and pretest 
survey items (see module 4) or to supplement information gathered from a survey. The goal 
of this module is to introduce participants to focus groups and to the considerations for 
planning, recruiting, and moderating them and for analyzing and reporting results. 

Strategies for adapting module 8 to your context 


• Module 8 is optional. If participants prefer to focus on survey development and 
administration, omit module 8 and instead conduct the activities in the other 
modules. If you skip this module, let participants know that they can find resourc- 
es about conducting focus groups on the Institute of Education Sciences website at 
http://ies.ed.gov/ncee/edlabs/projects/project.asp ?projectID=4544. 

• A second option is to use module 8 in a separate workshop for participants inter- 
ested in conducting focus groups. 

• You may choose to emphasize focus groups as a standalone research method for 
gathering information or as a way of formulating and pretesting survey items. If 
focusing on one aspect, inform participants that a focus group can serve both 
purposes. 


Ill 


Slides and notes for module 8 


Slide 1 
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Module 8 
presentation 
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Outline 


• Overview. 

• Planning. 

• Recruiting. 

• Moderating. 

• Analyzing and reporting. 
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Focus group results can be used to formulate research questions 
for survey projects and create survey items. When conducted early 
in the survey development process, focus groups can provide 
information about important aspects of the survey topic and 
the language that potential respondents use to describe their 
experiences and attitudes about the survey topic. 

In addition, focus groups can be used to pretest survey items (see 
module 4). Focus group feedback about an early draft of a survey 
can provide insight into how respondents may interpret and react 
to the survey items and how well the items cover all the relevant 
aspects of the survey topic. 


Slide 4 



Focus groups can also be used to gain more information about 
interesting research questions that arise from the survey findings. 
Focus groups can help researchers better understand why 
respondents may have chosen certain answers, and they can 
provide richer content or illustrative information about the topic 
covered in the survey. 

For example, if survey results indicate that many teachers are 
unhappy with a change in a district policy, a focus group may 
reveal more information about why teachers feel this way. The 
opportunity for participants to interact with each other can provide 
more intensive information than would be obtained through a 
series of individual interviews. 


Many mixed-methods studies use surveys and focus groups as 
complementary research methods. 
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Why focus groups? 


Slide 5 


This module explains how to conduct a focus group in the context 
of survey development. 




W 


Help formulate and pretest survey items. 
Explore quantitative survey findings. 
Use as a stand-alone research method. 


However, focus groups are also an effective research method 
outside the context of survey development, and the information 
here is relevant to standalone focus-group research efforts. 
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Examples of when to use 
focus groups 


Phases of focus group studies 

1. Planning. 

LaJ 

2. Recruiting. 

o [ 


4. Analyzing 

n- 

3. Moderating. 

and reporting 
results. 


Sample situations when the use of a focus group could be 
productive: 

• Your school is interested in implementing an afterschool 
program, and you want to get information from parents about 
their preferences, needs, concerns, and interest. 

• Your district has rolled out a new online student evaluation tool, 
and you want to get teachers’ feedback on how the tool is being 
used, problems with its implementation, and ideas for how it 
can be improved to support instruction. 

• A school community is concerned about an increase in bullying. 
District leaders want to understand the nature and extent of the 
problem and identify possible solutions from the viewpoints of 
students, parents, and school staff members. 

In each case, having a group discussion with stakeholder groups 
may yield different information from what a survey would. 
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Phases of focus group studies 
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Define research goals 


• How will the results will be 
used? 
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Define research goals 


• How will the results will be 
used? 

• What actions or decisions will 
the results of the study inform? 



Slide 11 


Define research goals 


• How will the results will be 
used? 


• What actions or decisions will 
the results of the study inform? 


What do you want to learn from 
the focus group study? 



Slide 12 



Carefully consider who should be represented in a focus group: 

• What are the important subgroups with common characteristics? 

• What subgroups might have different perspectives? Ensure that 
they are represented in your focus groups. 
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Determine group composition 


Each focus group typically has 6-10 members. Homogeneous 
grouping is typically better than heterogeneous grouping because: 

• Less time is required for participants to become comfortable 
with each other. 



• Group members will have a common style of communication. 

• Conversation is more likely to focus on the research topic. 

Be mindful of differences that may make focus group members 
feel less comfortable about sharing their thoughts. For example, 
if you want to know school administrators’ and teachers’ 
perspectives on a school reform initiative, it is better to have 
separate groups of administrators and teachers. Teachers 
may not feel comfortable expressing their views with local 
administrators in the room. 


Determine the number of groups 

• Three to five groups are 
sufficient. 


Determine the number of groups 

• Three to five groups are 
sufficient. 

• You will need a sufficient 
number of homogeneous groups 
for each subgroup. 


Determine the number of groups 

• Three to five groups are sufficient. 

• You will need a sufficient number 
of homogeneous groups for each 
subgroup. 

• Look for diminishing returns— 
more groups may not be 
necessary when you are not 
learning anything new. 


In general, three to five focus groups are sufficient; however, the 
number depends on the topic and questions of interest. 

For example, if you are interested in learning about parent, 
teacher, principal, and district administrator perceptions of a new 
statewide policy, you will likely want to conduct focus groups with 
different subgroups and in different geographic regions within the 
state. 


For example, if you are interested in student and teacher views 
on a district initiative, you will want to conduct at least two focus 
groups of students and two focus groups of teachers to capture 
enough information about each type of participant. 


When no new themes are emerging from a subgroup, no more 
focus groups are needed. 
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Focus group modes 


Slide 17 
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Prepare a moderator guide 


The guide 
can be 

structured or 
unstructured. 


Most focus groups are conducted in person, and this is generally 
the most effective format. However, groups also can be conducted 
in the following modes: 

Phone conference call. 

Internet webinar, with slides and audio. 

Computer-networked “chat-room” style groups, with typed 
moderator questions and typed participant responses (offers 
anonymity for sensitive issues). 

Whatever mode is used, specific directions regarding how to use 
technology (or participate in person) should be offered prior to 
beginning the focus group (Lavrakas, 2008; Morgan, 1997). 
Professional focus group moderators are highly skilled in working 
with people, but most are more experienced in consumer research 
than in education issues. 

Staff members can learn good moderator skills by reading 
literature about focus groups and participating in mock focus 
groups, in which participants give feedback to the moderator. 

A successful moderator has the following characteristics: 

• Is neutral. 

• Is a good listener. 

• Facilitates group discussion without leading or interjecting 
viewpoints. 

• Is able to engender trust. 

• Knows the topic and goals of the focus group. 

A moderator guide is similar to an interview protocol in that it 
lists questions that the moderator should ask the group and 
includes an introductory script and a list of prompts to facilitate 
conversation and probe further as needed (Morgan, 1997). 

The moderator guide can be unstructured or structured. 

An unstructured guide simply lists the topics. It allows the 
moderator more spontaneity and is better for exploratory studies. 

A structured guide has a list of specific questions and follow-up 
probes, and the moderator follows a script. Scripted focus groups 
allow more efficient analysis and are better for refining research 
questions. 


Phases of focus group studies 
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Slide 21 


1 Recruiting strategies 1 


• • 

L 1 J i i 



Recruiting strategies: 

• Select participants from a target population sampling frame (for 
example, parent or teacher lists). 

• Recruit participants through newsletters or emails. 

• Invite more than the desired number of participants to account 
for no-shows. 

• Offer monetary incentives if possible. Nonmonetary benefits, 
such as the opportunity to contribute to research about an 
important topic, also can be motivating. 

• Send multiple reminders to encourage attendance. 
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Phases of focus group studies 



Slide 23 



As focus group members arrive, be sure that they complete any 
documentation required to participate, such as an informed 
consent form (Morgan, 1997). 

The moderator should introduce the study and his or her role in it: 
the moderator is not an expert and is present only to facilitate the 
conversation. The goal of the focus group is for members to share 
and discuss their own perspectives. 

You may ask respondents to say something about themselves. 
This activity may ease nervousness among some members and 
break the ice. 

Common ground rules are: 


• Specify breaks. 

• State the confidentiality rules. 

• Note that only one person at a time may talk. 

• Urge respect for other opinions. 

• Start with a question that is applicable to all. 
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Moderator questions 


1 Question type 

Purpose 

Opening 

Helps participants get acquainted 
and feel connected 

Introductory 

Begins discussion of topic 

Transition 

Moves smoothly into key 
questions 

Key 

Obtains insight into areas of 
central concern to the study 

Ending 

Helps researchers determine 
where to place emphasis and 
brings closure to the discussion 


Moderator questions 
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Purpose 

Opening 

Helps participants get acquainted 
and feel connected 

Introductory 

Begins discussion of topic 

Transition 

Moves smoothly into key 
questions 

Key 

Obtains insight into areas of 
central concern to the study 

Ending 

Helps researchers determine 
where to place emphasis and 
brings closure to the discussion 


Moderator questions 


1 Question type 

Purpose 

Opening 

Helps participants get acquainted 
and feel connected 

Introductory 

Begins discussion of topic 

Transition 

Moves smoothly into key 
questions 

Key 

Obtains insight into areas of 
central concern to the study 

Ending 

Helps researchers determine 
where to place emphasis and 
brings closure to the discussion 


Opening questions. These questions are easy to answer and are 
designed to break the ice and make everyone feel comfortable. 

For example: 

In a focus group of parents: “Tell us your name and what grade 
your child is in.” 

In a focus group of students: “I’d like to take a few minutes for 
us to get to know each other. Instead of having you introduce 
yourselves the way it’s usually done, I’m going to ask you to 
introduce the person sitting next to you. Take a couple of minutes 
to talk to the person on your right. Find out his or her first name, 
age, grade in school, name of school, number of brothers and 
sisters, and what hobbies or afterschool activities the person is 
involved in” (Nolin & Chandler, 1996, p. 29). 

Introductory questions. These questions focus on the topic. The 
moderator asks about general impressions (the responses will 
give hints about participants’ experiences). For a student focus 
group on school safety, a good question is, "I’d like to start by 
talking about things that make you (or your friends) feel afraid or 
unsafe at school, such as things that may take place either while 
you are in school or going to or from school. What comes to mind 
first? What different things can you think of? (List on board.)” 

(Nolin & Chandler, 1996, p. 29). Probes about physical/verbal 
abuse, theft, vandalism, substance use, or locker crime, can be 
used as needed (Morgan & Krueger, 1998). 

Transition questions. These ask about experiences related to the 
topic, such as, “Let me bring up something else here: disruptive 
behavior. Does this go on at your school? What do you think 
of when I say “disruptive behavior?” What kinds of disruptive 
behaviors make the atmosphere at school seem unsafe or make it 
difficult for students to learn?” (Nolin & Chandler, 1996, p. 29). 

Key questions: 

• Ask targeted questions about the topic. 

• Ask follow-up questions. 

• Focus on specific questions, such as: 

° What did you do when...? 

° What prompted...? What happened...? 

° What would happen if...? 

° What would you change...? 

• Probe for clarity. 

Example: “Can you give me an example of something that 
happened at your school that made it seem unsafe — something 
that happened to you or one of your friends? How did this make 
you feel?” 

(Nolin & Chandler, 1996, p. 30) 

Ending questions. Participants are encouraged to summarize and 
clarify thoughts: 

• How would you describe your position on...? 

• Which. ..is most important? 

• What advice do you have about...? 

Example: “We’ve come to the end of the session. Is there 
anything you’d like to add on anything we’ve talked about?” 

(Nolin & Chandler, 1996, p. 33) 
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The moderator should not take notes during sessions. Plan for a 
note taker. 

The note taker should be familiar with the research goals of the 
study. Notes should include the following elements: 

• Quotations. 

• Key phrases word for word. 

• Major themes. 

• Areas of agreement and disagreement. 

• Other observations that may be relevant (for example, notable 
body language or tension between participants). 

Immediately after the focus group, even if the session has been 
recorded, the moderator should record impressions and general 
themes. It is imperative to do this immediately afterward so that 
recollection is not impeded by other experiences. (Morgan & 
Krueger, 1998). 

Audio recordings are recommended. If resources allow, make 
transcripts of the audio recordings. Video recording can be 
obtrusive and inhibit participation. 

If you audio or video record a focus group, include notice of the 
recording in the informed consent form that participants sign. In 
addition, participants should be explicitly told by the moderator 
at the beginning of the focus group that the session will be audio 
or video recorded. If any objections to recording are raised by any 
of the participants, recording should not take place (Morgan & 
Krueger, 1998). 
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Phases of focus group studies 
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Key steps in analysis: 

• Have multiple people review the transcript, recording, or notes. 

• Detect and summarize themes. 

• Note how many groups mentioned a topic, how often the topic 
was mentioned within the groups, and the agreement by group 
members. 

• Note differences in themes among subgroups. Record quotes 
that give evidence of each theme. 

(Morgan & Krueger, 1998) 
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Additional resources 


National Center for Education Statistics. (1996). 1993 National 

Household Education Survey (NHES:93) questionnaires: Screener, 
school readiness, and school safety and discipline (Working Paper 
96-21). Washington, DC: Author. Retrieved April 1, 2016, from 
http://nces.ed.gov/pubs96/9621.pdf . 


Sharing the results: 

• Recall the purpose of the study. 

• Consider the audience. Will the results be shared externally or 
only internally with the team? 

• Focus on clarity. Share common themes and note any 
differences in themes for different respondent types. 

• Avoid making statements that claim to represent a broader 
population. Do not write, “70 percent of teachers feel...”; 
instead write "Seven out of 10 participants mentioned....” 

• Link to decisions informed by findings. 

• Use quotations as examples of major themes. Quotations 
should not be attributed to individuals by name or associated 
with any other unique identifying feature. 

(Morgan & Krueger, 1998) 
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Additional resources 


For more information, please visit the following websites: 

• The American Association for Public Opinion Research. 
http://www.aapor.ora/ . 

• Public Opinion Quarterly: https://pog.oxfordiournals.org/ . 

• Journal of Survey Statistics and Methodology: 
http://issam.oxfordiournals.org/content/current . 
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Activity 8.1: Planning for a focus group 


After presenting the material in the slide deck, distribute handout 8.1. 

Encourage participants to work with a partner, particularly if they are working on the same 
project. Like activity 1.1, this activity can be completed quickly using a hypothetical focus 
group project, or it can take longer if the group is planning a real project. This activity also 
can be used as a resource document for participants outside the workshop. If you run out of 
time, encourage participants to complete the activity on their own. 

If you are completing this activity during the workshop, circulate among the participants 
to answer questions. Refer to the notes in the slide deck as a resource. 

If there is time at the end of the session, ask one or two individuals or groups to share their 
plan; then generate discussion and answer any questions from participants. 
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Handout 8.1: Planning for a focus group 


Think about a focus group project you will design in the near future- 

L What actions or decisions will the results of the study inform? What do you want 
the project to accomplish? What is your statement of purpose? What do you want 
the final report to include? 


2- What resources (money, time, and staff) are available for the project? 


3- Identify the target population- What are the important subgroups? Do you want 
homogeneous groups? Heterogeneous groups? Define the eligibility criteria- 


4. What is your recruitment strategy? What methods are likely to be most successful 
with this target population? How will the initial contact be made? 


5- How will you encourage attendance? What is the best location and time for these 
participants? 


6- Who will moderate the groups? 
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7. Begin outlining the moderator questions- 


8- How will data be collected? Who will take notes? Will a recording or a transcript 
be made? 


9- How will data be summarized? 
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Handout 8.2: Sample moderator guide for a 
structured focus group conducted to pretest survey items 


Introduction 

Thank you for agreeing to come. Your participation is very important. We are [self-intro- 
ductions for facilitator and note taker]. As you know, our state has a relatively new princi- 
pal evaluation system, and one feature of the system is that districts can collect survey data 
from teachers about principals and schools to provide data that can be used in principal 
evaluations. A draft survey will be piloted early next year. The current survey items were 
developed to align with the state s principal performance measures. 

Our discussion will focus on your reactions to the first draft of this survey and should last 
about 90 minutes. We will not take any breaks, but please feel free to get up at any time to 
stretch or use the restroom. Here are some ground rules for the discussion: 

• I am going to guide the discussion, asking the group a series of questions. [Note 
taker name] will be taking notes. We would like to hear from everybody, and I will 
encourage each of you to speak up. 

• There are no right or wrong answers, and it is okay to disagree with one another. 

• No one outside of our research team will have access to the notes from this focus 
group. All comments are confidential, and we will not identify anyone by name 
or their school in our reporting. Please keep in mind that information shared by 
your colleagues here today should be kept confidential by everyone in the room. 
Agreed? [Wait for nods.] Great, thank you. 

• This is a group discussion. You do not need to wait for me to call on you, but 
please speak one at a time to allow for accurate note-taking. 

• We have a lot of survey items to talk about, so I may cut short a conversation to 
move ahead so that we can cover everything. 

Before we begin, does anyone have any questions? 

Let s start by going around the room and introducing ourselves. Please share: 

• Your first name. 

• Three words to describe the school you work in. 

• One quick example of a positive experience, good piece of feedback, or surprising 
lesson learned from one of your principal evaluation processes. 

Focus group task orientation 

As you may know, our state has defined five measures of principal effectiveness. 

Display the measures and read them aloud . 

The survey development team created a survey to align with these five measures and 
their indicators. Not all aspects of principal effectiveness and leadership are suitable for a 
teacher survey. An effective principal does many things related to these measures that are 
either not well known by all teachers or are better measured through other data sources or 
stakeholders. This survey focuses on elements of these measures of principal effectiveness 
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that are knowable and easily reported by teachers and that can provide data that will be 
useful to principals and superintendents during the evaluation and feedback process. 

The survey development team’s goal is to create a survey that can be used to collect data 
from teachers that will be accurate and relevant and, therefore, useful to principals as they 
reflect on their work and leadership practices. Your honest feedback is welcome and needed 
for the survey development team to achieve this goal. 

Discuss survey items 

First let’s look at the indicators associated with performance measure 1: Establishes a 
vision for high student achievement supported by a culture of high expectations and family 
engagement. Take a moment to look through these handouts. 

Distribute handouts with indicators for this performance measure ♦ 

Now I will share the items that are being considered for the survey that relate to this 
measure. Take a moment to look over these survey items. 

Distribute handout with survey items associated with this measure ♦ 

Moderator questions ♦ After the survey draft is constructed, these questions may be mod- 
ified to highlight particular survey items about which the team is especially interested in 
hearing principal feedback or to generate a comparison and expressed preference between 
two similar survey items. It is unlikely that the moderator will use all 10 questions for each 
set of items, but they are here to guide the conversation as needed. 

1. Which of these survey items seem to you to align very well with this performance 
measure? Why? 

2. If you could select only five items from this set, which ones are the most 
important? 

3. Are there things you do at your school related to this performance measure — that 
teachers are aware of — that would not be captured by this set of survey items? 
Describe these things. 

4. Are any of these items asking teachers to provide information that you think they 
might not know or might not have an informed opinion about? 

5. Would principals who are very effective in this area get high ratings on these 
items from the teachers in their school? Why or why not? 

6. Would principals who are not effective in this area get low ratings on these items 
from the teachers in their school? Why or why not? 

7. Do you think you know how teachers in your school would answer these 
questions? 
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8. In what ways could your teachers’ answers to these items be useful to you and 
your work? 

9. Any other comments or suggestions for this set of items? 

10- How would you quickly summarize the feedback this group has for the survey 
development team about this set of survey items? How would you characterize 
your level of agreement about these recommendations? 

Repeat the cycle for the next four performance measures and related sets of survey items. 

Closing 

Thank you for sharing your views about this draft of the survey. Feedback from this focus 
group and information we are gathering from other principals, superintendents, and teach- 
ers will be used to revise this survey. A pilot test of the revised survey will be conducted 
next spring. Your contributions will help make this a survey that districts can use with 
confidence that it was created with the thoughtful input of principals. Thank you. 
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Appendix A. Resources for the workshop on 
survey methods in education research 

This appendix contains resources that are useful for planning and evaluating a survey methods workshop, 
including an agenda, a preworkshop survey, and a postworkshop evaluation survey. Each resource can be 
adapted to fit the structure and purpose of your workshop. 


Sample workshop agenda 


This sample is for a fulhday workshop covering all eight modules. This would be a lot of material to cover in 
one day for many audiences, so splitting this into two half-day sessions may be preferable. 

Time 

Session 

Topics 

Activities 

and handouts Presenter 

8:00-8:15 a.m. 

Introduction 



8:15-9:15 a.m. 

1. Planning for a survey 

Is a survey the right approach? 

Defining survey goals 

Writing research questions 

Defining topics and constructs 

Identifying subgroups and types of respondents 
Determining the unit of analysis 

Validity and reliability in surveys 

Activity 1.1 

Handout 1.1 

9:15-10:30 a.m. 

2. Exploring sources of 
existing survey items 

Why use existing surveys? 

Federal and other education survey sources 

Handout 2.1 

10:30-10:45 a.m. 

Break 



10:45-11:30 a.m. 

3. Writing survey items 

Purpose of survey items (questions) 

Writing clear items 

Open-ended and closed-ended items 

Response options 

Ratings scales and response options 

Formatting and layout 

Activity 3.1 

Handout 3.1 

11:30 a.m. -12:30 p.m. 

4. Pretesting survey items 

Expert reviews 

Cognitive interviews 

Focus groups 

Field testing 

Activity 4.1 

Handout 4.1 

12:30-1:00 p.m. 

Lunch 



1:00-1:45 p.m. 

5. Sampling 

Defining a target population 

Sampling frames 

Random sampling 

Stratification 

Census and other considerations 


1:45-2:30 p.m. 

6. Survey administration 

Personal interviews 

Online surveys 

Mailed surveys 


2:30-3:15 p.m. 

7. Response rates 

What are response rates? 

Calculating response rates 

Maximizing response rates 

Measuring nonresponse bias 

Handout 7.1 

Handout 7.2 

Handout 7.3 

3:15-3:30 p.m. 

Break 



3:30-4:15 p.m. 

8. Focus groups 

Purposes of focus groups 

Preparing protocols 

Choosing focus group members 

Moderator strategies 

Collecting and analyzing data 

Activity 8.1 

Handout 8.1 

Handout 8.2 

4:15-4:30 p.m. 

Closing and evaluations 
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Preworkshop needs assessment survey 


1. Have you ever taken a course in survey methodology? 

a. Yes 

b. No 

If yes, list the courses you have taken. 


2. Have you ever participated in the development of a survey? 

a. Yes 

b. No 

If yes, briefly describe the project(s) and your role(s). 


3. Have you ever participated in the administration of a survey? 

a. Yes 

b. No 

If yes, briefly describe the project(s) and your role(s). 


4. Have you ever participated in the analysis of survey results? 

a. Yes 

b. No 

If yes, briefly describe the project(s) and your role(s). 
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5- List any other relevant experience related to survey methodology that you have- 


6- What do you want to learn about conducting high-quality surveys? 


A-3 


Postworkshop evaluation survey 


Workshop on survey methods in education research 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Not 

applicable 

The goals for the workshop were clearly stated at or before the beginning of 
the event. 

□ 

□ 

□ 

□ 

□ 

The structure of the workshop was appropriate for meeting the stated goals. 

□ 

□ 

□ 

□ 

□ 

The presenter(s) explained the research evidence clearly. 

□ 

□ 

□ 

□ 

□ 

The presenter(s) clearly connected research evidence to practical 
implementation. 

□ 

□ 

□ 

□ 

□ 

Because of my participation, 1 have better knowledge of the basics of survey 
research. 

□ 

□ 

□ 

□ 

□ 

Because of my participation, 1 feel better equipped to carry out my 
organization’s research agenda and needs. 

□ 

□ 

□ 

□ 

□ 

Because of my participation, 1 feel that 1 can improve how my organization 
collects survey data. 

□ 

□ 

□ 

□ 

□ 

The format of the workshop provided sufficient time for the participants to 
meaningfully interact with each other. 

□ 

□ 

□ 

□ 

□ 

The workshop actively engaged me in learning the content. 

□ 

□ 

□ 

□ 

□ 

The workshop was relevant to an issue currently facing my organization. 

□ 

□ 

□ 

□ 

□ 

The presenter was knowledgeable or experienced with the type of setting or 
role in which 1 work. 

□ 

□ 

□ 

□ 

□ 

The workshop provided opportunities to consider how to use research 
or effectively incorporate data into decisionmaking within my agency or 
organization. 

□ 

□ 

□ 

□ 

□ 

1 expect to apply information from the workshop in my work. 

□ 

□ 

□ 

□ 

□ 

1 expect to share the information 1 learned at the workshop with my 
colleagues. 

□ 

□ 

□ 

□ 

□ 

If additional workshops or trainings were offered, 1 would be likely to attend. 

□ 

□ 

□ 

□ 

□ 

1 am satisfied with the overall quality of this workshop. 

□ 

□ 

□ 

□ 

□ 

The benefits of attending this workshop were worth the time 1 invested. 

□ 

□ 

□ 

□ 

□ 


18- What aspects of this workshop were most helpful and why? 


19. What aspects of this workshop were least helpful and why? 
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20 . What additional follow-up activities would help you increase your knowledge of 
this topic or apply the information to your own work? 


21. What part of this workshop would you suggest changing to make it better for 
future participants? 


22. Asa result of this workshop, what action steps will you take? 
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Appendix B. Glossary of terms and additional resources for 
the workshop on survey methods in education research 


Census. A form of data collection in which every member, rather than a sample, of a 
target population is surveyed- 

Closed- ended item. An item that requires respondents to choose from a predetermined 
set of options, such as a point on a scale that ranges from strongly agree to strongly disagree . 

Cognitive interview. A one-on-one interview designed to evaluate measurement error in 
survey items- Individuals selected to participate in a cognitive interview should be similar 
to the respondents of interest. 

Collectively exhaustive. A desirable characteristic of a set of responses for a closed-ended 
item whereby the response categories fully cover the range of possible responses. 

Construct. A conceptual idea that can be measured through one or multiple survey 
items- A construct may have multiple subcomponents- For example, the construct “parent 
involvement” includes supporting student learning at home, volunteering in the classroom, 
communicating with teachers, and other components- A subcomponent may be measured 
through one item or a series of survey items. 

Convenience sampling. A method of sampling in which the researcher surveys a group 
of individuals who are readily available or easily contacted (for example, attending a par- 
ent-teacher association meeting)- These kinds of samples are not considered representative 
of the target population, although they may be useful for some pretesting activities. 

Double-barreled item. An item that contains two separate concepts (for example, “Do you 
agree the principal has a strong leadership style and is respected?”)- Double-barreled items 
should be avoided. 

Expert review. Consultation with researchers or practitioners who have experience in and 
knowledge of a particular topic area to inform research goals, including issues related to 
data collection and what survey items should be asked. 

Field test. A small-scale administration of a survey designed to mirror the data collection 
procedures of the full-scale study. The purpose of a field test is to evaluate administration 
procedures and survey items to identify potential problems before commencing full-scale 
data collection. Often used in high-stakes surveys, field testing can lead to revisions that 
improve the survey with respect to administration procedures, clarity, completeness of 
data, and accuracy. 

Focus group. A qualitative research method in which a moderator facilitates a group 
interview and discussion among 6-10 individuals on a particular topic. Focus groups can 
be used to generate research questions, formulate and pretest potential survey items, or 
supplement the information gathered from surveys. 
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Measurement error. The difference between a measured value of a quantity or construct 
and its true value. Measurement error can be caused by several factors, including missing 
responses, lack of clarity, omitted items that address a subcomponent of the construct, and 
errors in data collection and recording. 

Mode. The method by which respondents complete a survey (for example, telephone, web, 
or mail). 

Mutually exclusive. A desirable characteristic of a set of closed-ended responses for a 
survey item whereby each category is distinct and only one response could be true. 

Nonresponse bias. Bias in survey findings that occurs when the individuals who did not 
respond to a survey item differ from the individuals who did respond with respect to how 
they would answer the items. For example, if participants are asked to rate their experienc- 
es with a program and individuals who liked the program are more likely to respond to the 
survey, the survey results will contain bias and be misleading. 

Open-ended item. An item in which respondents are given space to construct their own 
response rather than choose from a list of provided responses (closed-ended item). 

Pretesting. The process of evaluating whether items are likely to capture the desired infor- 
mation from respondents. Pretesting can occur in one of four primary ways: expert review, 
cognitive interviews, focus groups, and field testing. 

Random sampling. A sampling method in which a subset of the target population is 
selected at random, where each member has a known chance of being selected. A random 
sample of sufficient size will elicit approximately the same overall responses as a census of 
the target population. 

Rating scale. A response option set of ordered categories indicating the degree of some 
characteristic, such as importance (for example, not important, somewhat important, import- 
ant, and very important ). 

Reliability. The consistency with which a set of items measures an underlying construct 
over repeated administrations (or within an administration). A reliable survey instrument 
has items that produce stable results about its constructs. 

Respondent. An individual who responds to a survey or questionnaire. 

Response rate. The percentage of people who complete and return the survey out of the 
total number of surveys intended to be completed. 

Retrospective probing. A strategy used during cognitive interviews in which the inter- 
viewer asks the respondent specific questions after the respondent has answered a survey 
item or set of survey items. 

Sample. A subset of the target population from which data will be collected. Several 
methods can be used to select a sample, including convenience sampling, random sam- 
pling, and stratified random sampling. 
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Sampling frame. A complete list of all eligible members of a target population. 

Scale. A set of survey items (typically three or more) that measure a construct. 

Scripted probe. In a cognitive interview, a question asked by all interviewers from an 
interview protocol. 

Spontaneous probe. In a cognitive interview, a question asked as a follow-up to a partici- 
pant’s response to an item that was not scripted. 

Stratified random sampling. A sampling method in which the target population is divided 
into groups, or strata, based on a characteristic or set of characteristics (for example, age or 
location), and then a random sample is taken from each group. Stratified random sampling 
often is used to increase representativeness of the random sample by controlling its align- 
ment to population characteristics. 

Survey methodology. The science of survey design, data collection, processing, and 
analysis. 

Target population. The group of people or organizations that the survey results will rep- 
resent and the group of individuals or organizations from which a sample is drawn or a 
census is conducted. 

Think-aloud. A strategy used during cognitive interviews in which the interviewer asks 
respondents to describe what they are thinking as they answer survey items. This strategy 
allows the interviewer to understand respondents’ cognitive processes while answering the 
item and reveals items that are not interpreted as expected. 

Unit of analysis. The people, programs, or institutions the researcher wants to make state- 
ments about. 

Validity. The extent to which the inferences made on the basis of survey responses are 
accurate and meaningful. A valid survey instrument provides results that accurately reflect 
the measured constructs and the target population as intended by the survey designers and 
audiences of the survey findings. 
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Note 


1. See, for example, the YouTube video of a webinar for a survey development project 
with REL Midwest’s Early Childhood Education Research Alliance (https://youtu.be/ 
2y0RxCvfW10?list=PLVHqsnePfULp01KpPD_f6Y6JRH4i4M00x). 


Notes-1 


References 


American Association for Public Opinion Research. (2009a). Best practices for research. 
Deerfield, IL: Author. Retrieved April 1, 2016, from http://www.aapor.org/Standards 
-Ethics/Best-Practices, aspx. 

American Association for Public Opinion Research. (2009b). Standard definitions: Final 
dispositions of case codes and outcome rates for surveys (6th edition). Deerfield, IL: Author. 

Burns, S., Wang, X., & Henning, A. (2011). NCES handbook of survey methods (NCES 
No. 2011-609). Washington, DC: U.S. Department of Education, Institute of Educa- 
tion Sciences, National Center for Education Statistics. Retrieved April 1, 2016, from 
http://nces.ed.gov/pubs2011/2011609.pdf 

Dillman, D. A. (2000). M ail and internet surveys: The total design method (2nd edition). New 
York, NY: Wiley. 

Fowler, Jr., F. J. (2009). Survey research methods. In L. Brickman & D. J. Rog (Eds.), Applied 
social research methods series: Vol. 1. (pp. 19-45). Newbury Park, CA: SAGE. 

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. 
Public Opinion Quarterly, 70(5), 646-675. 

Jabine, T. B., Straf, M. L., Tanur, J. M., & Tourangeau, R. (eds.). (1984). Cognitive aspects 
of survey methodology: Building a bridge between disciplines. Washington, DC: National 
Academy Press. 

Kendziora, K. K., & Boccanfuso, C. (2011). Survey management (Webinar series for the 
National Center on Safe Supportive Learning Environments). Retrieved April 1, 2016, 
from http://safesupportivelearning.ed.gov/events/webinar/survey-management. 

Klein, L. (2008). Modes of survey data collection. Falls Church, VA: NRI. Retrieved Decem- 
ber 27, 2016, from http://media.wix.com/ugd/186708_9eel87083cba49c381f8ce0a95acf- 
c2c.pdf. 

Lavrakas, P. J. (2008). Encyclopedia of survey research methods. Thousand Oaks, CA: SAGE. 

Morgan, D. L. (1997). Focus groups as qualitative research (2nd edition). Thousand Oaks, 
CA: SAGE. 

Morgan D. L., & Krueger, R. A. (1998). The focus group kit. Thousand Oaks, CA: SAGE. 

Nolin, M. J., & Chandler, K. (1996). Use of cognitive laboratories and recorded interviews 
in the National Household Education Survey (NCES 96-332). Washington, DC: U.S. 
Department of Education, National Center for Education Statistics. 

Pastor, D. A. (2013). Considerations in evaluating, selecting, and developing data sources and 
measures (Unpublished webinar for the Regional Educational Laboratories). 


Ref-1 


Ruddy, S. (2011). Survey administration (Webinar series for the National Center on 
Safe Supportive Learning Environments). Retrieved April 1, 2016, from http:// 
safesupportivelearning.ed.gov/events/webinar/survey-administration. 

Scheuren, F. (n.d.). What is a survey? Alexandria, VA: American Statistical Association. 
Retrieved April 1, 2016, from http://www.whatisasurvey.info/. 

Thurgood, L., Walter, E., Carter, G. Henn, S., Huang, G. Nooter, D., et al. (2003). NCES 
handbook of survey methods: Technical report (NCES No. 2003-603). Washington, DC: 
U.S. Department of Education, Institute of Education Sciences, National Center for 
Education Statistics. http://eric.ed.gov/7kHED480803 

Tucker, C., & Medway, R. (2013). Survey methods . Presentation to American Institutes for 
Research staff, Washington, DC. 

Willis, G. B. (1999). Cognitive interviewing : A “how-to” guide. Rockville, MD: Research Tri- 
angle Institute. Retrieved April 1, 2016, from http://appliedresearch.cancer.gov/archive/ 
cognitive/interview.pdf. 

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design . Thou- 
sand Oaks, CA: SAGE. 


Additional resources 


These additional resources provide more in-depth information about the topics presented 
in the workshop. 

Buckingham, A., & Saunders, P. (2004). The survey methods workbook: From design to 
analysis. Cambridge, UK: Polity. 

Converse, J. M., & Presser, S. (1986). Survey questions: Handcrafting the standardized ques- 
tionnaire. Newbury Park, CA: SAGE. 

Couper, M. (2008). Designing effective web surveys. Cambridge, UK: Cambridge University 
Press. 

Excel 2010 Statistics 66: Simple random sample using RAND function, sort and paste 
(Video). Retrieved April 1, 2016, from http://www.youtube.com/watch?v=Q5gB3qX0z-E. 

Groves, R. M. (1989). Survey errors and survey costs. New York, NY: Wiley. 

Groves, R. M., Fowler, F. J., Couper, M., Lepkowski, J. M., Singer, E., & Tourangeau, R. 
(2009). Survey methodology. Hoboken, NJ: Wiley. 

Hansen, M. H., Hurwitz, W. N., & Madow, W. G. (1953). Sample survey methods and theory. 
New York, NY: Wiley. 

Kalton, G. (1986). Introduction to survey sampling. Thousand Oaks, CA: SAGE. 


Ref-2 


Kish, L, (1965), Survey sampling. New York, NY: Wiley, 

Payne, S. L, (1951), The art of asking questions. Princeton, NJ: Princeton University Press, 

Pfefferman, D,, & Rao, C, R, (2009), Sample surveys: Design, methods and applications , 
Amsterdam, Netherlands: Elsevier/North-Holland. 

Schuman, H,, & Presser, S, (1981), Questions and answers in attitude surveys. Cambridge, 
UK: Academic Press, 

Shindler, J, V. (2011), Survey development. (Webinar series for the National Center on 
Safe Supportive Learning Environments), Retrieved April 1, 2016, from http:// 
safesupportivelearning,ed,gov/events/webinar/survey-development, 

Sudman, S,, & Bradburn, N, M. (1974)- Response effects in surveys. Chicago, IL: Aldine, 
Retrieved April 1, 2016, from http://www,norc,org/PDFs/Publications/Response_ 
Effects_in_Surveys_1974.pdf. 

U,S, Department of Education, (2016), Common education data standards. Retrieved April 
1, 2016, from https://ceds,ed.gov/. 


Ref-3 


The Regional Educational Laboratory Program produces 7 types of reports 



Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 


What’s Known 

Summaries of previous research 


Stated Briefly 

Summaries of research findings for specific audiences 


Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 



